Order Matters in the Presence of Dataset Imbalance for Multilingual Learning
Dami Choi, Derrick Xin, Hamid Dadkhahi, Justin Gilmer, Ankush Garg, Orhan Firat, Chih-Kuan Yeh, Andrew M. Dai
Abstract
In this paper, we empirically study the optimization dynamics of multi-task learning, particularly focusing on those that govern a collection of tasks with significant data imbalance. We present a simple yet effective method of pre-training on high-resource tasks, followed by fine-tuning on a mixture of high/low-resource tasks. We provide a thorough empirical study and analysis of this method's benefits showing that it achieves consistent improvements relative to the performance trade-off profile of standard static weighting. We analyze under what data regimes this method is applicable and show its improvements empirically in neural machine translation (NMT) and multi-lingual language modeling.
BibTeX
@inproceedings{
choi2023order,
title={Order Matters in the Presence of Dataset Imbalance for Multilingual Learning},
author={Dami Choi and Derrick Xin and Hamid Dadkhahi and Justin Gilmer and Ankush Garg and Orhan Firat and Chih-Kuan Yeh and Andrew M. Dai and Behrooz Ghorbani},
booktitle={Thirty-seventh Conference on Neural Information Processing Systems},
year={2023},
url={https://openreview.net/forum?id=7RMGI4slcb}
}