Zhenxin Qin

A Unified Approach to Interpreting Knowledge Distillation for Large Language Models via Interactions

Despite the success of knowledge distillation (KD) in Large Language Models (LLMs), the underlying mechanism behind its efficacy remains unclear. In this paper, we propose a unified approach to explore the common mechanism of various KD methods using interactions. Specifically, we decompose the outp…