2026
ExpertWeaver: Unlocking the Inherent MoE in Dense LLMs with GLU Activation Patterns
ICML 2026poster
Mixture-of-Experts (MoE) effectively scales model capacity while preserving computational efficiency through sparse expert activation. However, training high-quality MoEs from scratch is prohibitively expensive. A promising alternative is to convert pretrained dense models into sparse MoEs. Existing…