Nhat-Tri Ho

Geometric and Stochastic Analysis of Discontinuities in Sparse Mixture-of-Experts

Sparse Mixture-of-Experts (SMoE) architectures are now widely deployed in state-of-the-art language and vision models, where conditional routing allows scaling to very large networks. However, this very Top-$k$ expert selection that enables conditional routing also renders the SMoE map inherently di…