Sophia I. Thomopoulos

ERMoE: Eigen-Reparameterized Mixture-of-Experts for Stable Routing and Interpretable Specialization

Mixture-of-Experts (MoE) models expand capacity via sparse expert activation, but routing logits can misalign with expert structure (unstable routing, underutilization) and load imbalance can create stragglers. Auxiliary load-balancing losses reduce disparity but often weaken specialization and down