Puneeth S.M. Saladi

Pro-KD: Progressive Distillation by Following the Footsteps of the Teacher

With the ever growing scale of neural models, knowledge distillation (KD) attracts more attention as a prominent tool for neural model compression. However, there are counter intuitive observations in the literature showing some challenging limitations of KD. A case in point is that the best perform…