2026
Efficient Bilevel Optimization for CKA-Guided MoE Upcycling
ICML 2026poster
Upcycling, a strategy that initializes Mixture-of-Experts (MoE) by replicating pre-trained feed-forward or MoE networks to expand model capacity, has become a popular method in continual learning due to its effectiveness in mitigating catastrophic forgetting. However, existing paradigms rely on indi…