2022
Efficient Large Scale Language Modeling with Mixtures of Experts
EMNLP 2022main
Mixture of Experts layers (MoEs) enable efficient scaling of language models through conditional computation. This paper presents a detailed empirical study of how autoregressive MoE language models scale in comparison with dense models in a wide range of settings: in- and out-of-domain language mod…