Isotropic Representation Can Improve Zero-Shot Cross-Lingual Transfer on Multilingual Language Models
Yixin Ji, Jikai Wang, Juntao Li, Hai Ye, Min Zhang
Abstract
With the development of multilingual pre-trained language models (mPLMs), zero-shot cross-lingual transfer shows great potential. To further improve the performance of cross-lingual transfer, many studies have explored representation misalignment caused by morphological differences but neglected the misalignment caused by the anisotropic distribution of contextual representations. In this work, we propose enhanced isotropy and constrained code-switching for zero-shot cross-lingual transfer to alleviate the problem of misalignment caused by the anisotropic representations and maintain syntactic structural knowledge. Extensive experiments on three zero-shot cross-lingual transfer tasks demonstrate that our method gains significant improvements over strong mPLM backbones and further improves the state-of-the-art methods.\footnote{Our code will be available at \url{https://github.com/Dereck0602/IsoZCL}.}
BibTeX
@inproceedings{
ji2023isotropic,
title={Isotropic Representation Can Improve Zero-Shot Cross-Lingual Transfer on Multilingual Language Models},
author={Yixin Ji and Jikai Wang and Juntao Li and Hai Ye and Min Zhang},
booktitle={The 2023 Conference on Empirical Methods in Natural Language Processing},
year={2023},
url={https://openreview.net/forum?id=UlgNWOzMz2}
}