Seonyeong Heo

What to Distill? Fast Knowledge Distillation with Adaptive Sampling

Knowledge Distillation (KD) has been established as an effective technique for reducing the resource requirements of models when tackling computer vision tasks. Prior work has studied how to distill the knowledge of a teacher model better, but it overlooks how data affects the distillation result. T…