Guichao Zhu

FoldMoE: Efficient Long Sequence MoE Training via Attention-MoE Pipelining

Training LLMs with Mixture-of-Experts (MoE) architecture on long sequences poses significant challenges due to the all-to-all communication bottleneck of expert parallelism. While existing approaches attempt to hide the communication costs in computation through token-level pipelining within MoE lay…