Kaiwen Kacuila

The Bridge-Garden Dilemma in LLM Distillation: Why Mixing Hard and Soft Labels Works

Knowledge distillation (KD) transfers knowledge from a large teacher model to a smaller student. In language modeling, the student is trained either on tokens sampled from the teacher (\textbf{hard labels}) or the teacher’s full next-token distribution (\textbf{soft labels}). Despite soft labels app…