Class-wise Adaptive Logits Distillation with Meta-Learning

Abstract

Knowledge distillation has become a crucial technique for transferring intricate knowledge from a teacher model to a smaller student model. While logit-based knowledge distillation has shown promise, existing methods often overlook the efficient distillation of logits. In this paper, we introduce a novel approach called Class-wise Adaptive Logits Distillation (CALD) based on meta-learning. Our method leverages a meta-network to generate class-adaptive weights, delivering both explicit and implicit knowledge adaptively. By training the meta-network to assign higher weights to specific classes crucial for the student model’s learning from the teacher model, our approach enhances the knowledge transfer process. Experimental results on CIFAR-100 and ImageNet datasets demonstrate that CALD surpasses state-of-the-art knowledge distillation methods, achieving enhanced accuracy and efficiency in transferring knowledge from teacher to student models.