AAAI 2026technical0 citations
SMIDT: High-Performance Inference Framework for MoE Models with Dynamic Top-K Routing
Zewen Jin, Shen Fu, Chengjie Tang, Youhui Bai, Shengnan Wang, Jiaan Zhu, Chizheng Fang, Ping Gong
Abstract
To accelerate Mixture-of-Experts (MoE) inference, the hybrid parallelism paradigm is first applying pipeline parallelism (PP) to vertically divide the model into stages, with each stage further divided horizontally using tensor or expert parallelism. On the algorithm side, dynamic Top-K routing reduces computation by activating fewer experts per token on average. In this paper, we explore the application of dynamic Top-K routing to PP-enabled MoE inference, aiming to fully unleash their combined potential. We identify key performance bottlenecks arising from Top-K value variation across layers, which conflicts with PP
BibTeX
@inproceedings{aaai2026_smidthighperform,
title = {SMIDT: High-Performance Inference Framework for MoE Models with Dynamic Top-K Routing},
author = {Zewen Jin and Shen Fu and Chengjie Tang and Youhui Bai and Shengnan Wang and Jiaan Zhu and Chizheng Fang and Ping Gong and Cheng Li},
booktitle = {AAAI 2026},
year = {2026}
}