Fuyu Lv

SERE: Similarity-based Expert Re-routing for Efficient Batch Decoding in MoE Models

Mixture-of-Experts (MoE) architectures employ sparse activation to deliver faster training and inference with higher accuracy than dense LLMs. However, in production serving, MoE models require batch inference to optimize hardware efficiency, which may cause excessive expert activation and thus slow…