2024
Mixture of Tokens: Continuous MoE through Cross-Example Aggregation
NeurIPS 2024poster
Mixture of Experts (MoE) models based on Transformer architecture are pushing the boundaries of language and vision tasks. The allure of these models lies in their ability to substantially increase the parameter count without a corresponding increase in FLOPs. Most widely adopted MoE models are disc…