2026
MoSE: Hierarchical Self-Distillation Enhances Early Layer Embeddings
AAAI 2026technical
Deploying language models often requires navigating accuracy vs. performance trade-offs to meet latency constraints while preserving utility. Traditional model distillation reduces size but incurs substantial costs through training separate models. We introduce ModularStarEncoder (MoSE), a 1-billion