2024
Scaling Beyond the GPU Memory Limit for Large Mixture-of-Experts Model Training
ICML 2024poster
Mixture-of-Experts (MoE) is a powerful technique for enhancing the performance of neural networks while decoupling computational complexity from the number of parameters. However, despite this, scaling the number of experts requires adding more GPUs. In addition, the load imbalance in token load acr…