Hong-Xia Xie

Perspective-Aware Teaching: Adapting Knowledge for Heterogeneous Distillation

Knowledge distillation (KD) involves transferring knowledge from a pre-trained heavy teacher model to a lighter student model, thereby reducing the inference cost while maintaining comparable effectiveness. Prior KD techniques typically assume homogeneity between the teacher and student models. Howe…