2024
Pushing Mixture of Experts to the Limit: Extremely Parameter Efficient MoE for Instruction Tuning
ICLR 2024poster
The Mixture of Experts (MoE) is a widely known neural architecture where an ensemble of specialized sub-models optimizes overall performance with a constant computational cost. However, conventional MoEs pose challenges at scale due to the need to store all experts in memory. In this paper, we push…