Shayan Mohajer Hamidi*

How to Train the Teacher Model for Effective Knowledge Distillation

"Recently, it was shown that the role of the teacher in knowledge distillation (KD) is to provide the student with an estimate of the true Bayes conditional probability density (BCPD). Notably, the new findings propose that the student’s error rate can be upper-bounded by the mean squared error (MSE…