Cross-lingual Retrieval for Iterative Self-Supervised Training
Chau Tran, Yuqing Tang, Xian Li, Jiatao Gu
Abstract
Recent studies have demonstrated the cross-lingual alignment ability of multilingual pretrained language models. In this work, we found that the cross-lingual alignment can be further improved by training seq2seq models on sentence pairs mined using their own encoder outputs. We utilized these findings to develop a new approach --- cross-lingual retrieval for iterative self-supervised training (CRISS), where mining and training processes are applied iteratively, improving cross-lingual alignment and translation ability at the same time. Using this method, we achieved state-of-the-art unsupervised machine translation results on 9 language directions with an average improvement of 2.4 BLEU, and on the Tatoeba sentence retrieval task in the XTREME benchmark on 16 languages with an average improvement of 21.5% in absolute accuracy. Furthermore, CRISS also brings an additional 1.8 BLEU improvement on average compared to mBART, when finetuned on supervised machine translation downstream tasks.
BibTeX
@inproceedings{NEURIPS2020_1763ea5a,
author = {Tran, Chau and Tang, Yuqing and Li, Xian and Gu, Jiatao},
booktitle = {Advances in Neural Information Processing Systems},
editor = {H. Larochelle and M. Ranzato and R. Hadsell and M.F. Balcan and H. Lin},
pages = {2207--2219},
publisher = {Curran Associates, Inc.},
title = {Cross-lingual Retrieval for Iterative Self-Supervised Training},
url = {https://proceedings.neurips.cc/paper_files/paper/2020/file/1763ea5a7e72dd7ee64073c2dda7a7a8-Paper.pdf},
volume = {33},
year = {2020}
}