Renhao Tan — accepted AI-conference papers

2024

Bayes Conditional Distribution Estimation for Knowledge Distillation Based on Conditional Mutual Information

Linfeng Ye, Shayan Mohajer Hamidi, Renhao Tan, EN-HUI YANG

ICLR 2024poster

It is believed that in knowledge distillation (KD), the role of the teacher is to provide an estimate for the unknown Bayes conditional probability distribution (BCPD) to be used in the student training process. Conventionally, this estimate is obtained by training the teacher using maximum log-like…

Cited by 17 Source ↗PDF ↗code ↗Scholar ↗

2024

How to Train the Teacher Model for Effective Knowledge Distillation

Shayan Mohajer Hamidi*, Xizhen Deng, Renhao Tan, Linfeng Ye, Ahmed Hussein Salamah

ECCV 2024poster

"Recently, it was shown that the role of the teacher in knowledge distillation (KD) is to provide the student with an estimate of the true Bayes conditional probability density (BCPD). Notably, the new findings propose that the student’s error rate can be upper-bounded by the mean squared error (MSE…

Cited by 4 Source ↗PDF ↗code ↗Scholar ↗