Bryan Gopal

Opportunistic Expert Activation: Batch-Aware Expert Routing for Faster Decode Without Retraining

An increasing number of LLMs employ Mixture-of-Experts (MoE) architectures where the feed-forward layer is replaced by a pool of experts and each token only activates a small subset of them. During autoregressive generation, these models often enter a memory-bound regime even for moderate batch size…