Venmugil Elango

Revisiting Efficiency–Accuracy Scaling in Mixture-of-Experts Architectures

Mixture-of-Experts (MoEs) have become a central component of many state-of-the-art open-source and proprietary large language models. Despite their widespread adoption, it remains unclear how close existing MoE architectures are to optimal with respect to inference cost, as measured by accuracy per …