2025
Beyond Logits: Aligning Feature Dynamics for Effective Knowledge Distillation
ACL 2025long
Knowledge distillation (KD) compresses large language models (LLMs), known as teacher models, into lightweight versions called student models, enabling efficient inference and downstream applications. However, prevailing approaches accomplish this by predominantly focusing on matching the final outp…