2024
BAM! Just Like That: Simple and Efficient Parameter Upcycling for Mixture of Experts
NeurIPS 2024poster
Mixture of Experts (MoE) framework has become a popular architecture for large language models due to its superior performance compared to dense models. However, training MoEs from scratch in a large-scale regime is prohibitively expensive. Previous work addresses this challenge by independently tra…