Mansour ZOUBEIROU A MAYAKI

Generalization and Scaling Laws for Mixture-of-ExpertsTransformers

We develop a theory of generalization and scaling for Mixture-of-Experts (MoE) Transformers that cleanly separates active per-input capacity from routing combinatorics. Conditioning on fixed routing patterns and union-bounding across them, we obtain a sup-norm covering-number bound whose metric entr…