Hamza ElMokhtar Shili

Learning to Specialize: Joint Gating-Expert Training for Adaptive MoEs in Decentralized Settings

Mixture-of-Experts (MoEs) achieve scalability by dynamically activating subsets of their components. Yet, understanding how expertise emerges through joint training of gating mechanisms and experts remains incomplete, especially in scenarios without clear task partitions. Motivated by inference cost…