Jo{\~a}o Maria Janeiro

Mixture of Languages: Improved Multilingual Encoders Through Language Grouping

We propose Mixture of Languages (MoL), a new strategy to pretrain largely multilingual encoders. Recent work in this field has relied on training transformer encoders on a large amount of multilingual data, with all parameters shared across all languages, without studying how to optimally balance la