JunhuaShu JunhuaShu

Beyond Logits: Aligning Feature Dynamics for Effective Knowledge Distillation

Knowledge distillation (KD) compresses large language models (LLMs), known as teacher models, into lightweight versions called student models, enabling efficient inference and downstream applications. However, prevailing approaches accomplish this by predominantly focusing on matching the final outp…