Federico Pennino

MoSE: Hierarchical Self-Distillation Enhances Early Layer Embeddings

Deploying language models often requires navigating accuracy vs. performance trade-offs to meet latency constraints while preserving utility. Traditional model distillation reduces size but incurs substantial costs through training separate models. We introduce ModularStarEncoder (MoSE), a 1-billion