Jung Jun Lee

Mixture-of-Clustered-Experts: Advancing Expert Specialization and Generalization in Instruction Tuning

A sparse Mixture-of-Experts (MoE) architecture has emerged as a highly scalable solution by conditionally activating sub-modules without a proportional increase in computational costs. However, improving expert specialization to enhance performance and generalization remains a challenge for MoE, esp