Yang Ya

KD-Zero: Evolving Knowledge Distiller for Any Teacher-Student Pairs

Knowledge distillation (KD) has emerged as an effective technique for compressing models that can enhance the lightweight model. Conventional KD methods propose various designs to allow student model to imitate the teacher better. However, these handcrafted KD designs heavily rely on expert knowle…