SIMoE: A Probabilistic Framework for Cardinality-Constrained Routing in Mixture-of-Experts
Mixture-of-Experts (MoE) models scale by activating only a small subset of experts per token, but standard deterministic top-$k$ routing is non-differentiable and trained using surrogate gradients that ignore the discrete expert selection used at inference. We introduce SIMoE routing by modeling exp…