RCMoE: A Communication-Efficient Random Compression Framework for Resource-Constrained Mixture-of-Experts Training
Donglei Wu, Xiao Cai, Jinglei Tan, Jinda Jia, Guangming Tan, Dingwen Tao, Wen Xia, Zhihong Tian
Abstract
Mixture-of-Experts (MoE) architecture with experts parallelism scales LLMs efficiently by activating only a subset of experts per input, avoiding proportional training costs. However, the intensive and heterogeneous communication substantially hinders the efficiency and scalability of MoE training in the resource-constrained scenario. Existing communication compression techniques fall short in MoE training due to: (i) Intensive training amplifies compression overhead, compromising training efficiency; (ii) Accumulated compression errors propagate through the network, degrading training quality. In this paper, we propose RCMoE, a communication-efficient Random Compression framework for MoE training with two core modules: (1) Local-Stochastic Quantization compresses the all-to-all communication by stochastically quantizing each row of the expert
BibTeX
@inproceedings{aaai2026_rcmoeacommunicat,
title = {RCMoE: A Communication-Efficient Random Compression Framework for Resource-Constrained Mixture-of-Experts Training},
author = {Donglei Wu and Xiao Cai and Jinglei Tan and Jinda Jia and Guangming Tan and Dingwen Tao and Wen Xia and Zhihong Tian},
booktitle = {AAAI 2026},
year = {2026}
}