Part-Of-Speech Sensitivity of Routers in Mixture of Experts Models

Elie Antoine; Frederic Bechet; Phillippe Langlais

COLING 2025main1 citations

Part-Of-Speech Sensitivity of Routers in Mixture of Experts Models

Elie Antoine, Frederic Bechet, Phillippe Langlais

Abstract

This study investigates the behavior of model-integrated routers in Mixture of Experts (MoE) models, focusing on how tokens are routed based on their linguistic features, specifically Part-of-Speech (POS) tags. The goal is to explore across different MoE architectures whether experts specialize in processing tokens with similar linguistic traits. By analyzing token trajectories across experts and layers, we aim to uncover how MoE models handle linguistic information. Findings from six popular MoE models reveal expert specialization for specific POS categories, with routing paths showing high predictive accuracy for POS, highlighting the value of routing paths in characterizing tokens.

BibTeX

@inproceedings{antoine-etal-2025-part,
    title = "Part-Of-Speech Sensitivity of Routers in Mixture of Experts Models",
    author = "Antoine, Elie  and
      Bechet, Frederic  and
      Langlais, Phillippe",
    editor = "Rambow, Owen  and
      Wanner, Leo  and
      Apidianaki, Marianna  and
      Al-Khalifa, Hend  and
      Eugenio, Barbara Di  and
      Schockaert, Steven",
    booktitle = "Proceedings of the 31st International Conference on Computational Linguistics",
    month = jan,
    year = "2025",
    address = "Abu Dhabi, UAE",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2025.coling-main.431/",
    pages = "6467--6474"
}