BDCKD: Unlocking the Power of Brownian Distance Covariance in Knowledge Distillation

Abstract

Knowledge distillation has been proven to be an effective method for enhancing model performance, particularly in the domain of model compression. In this study, we propose a comprehensive approach that utilizes Brownian Distance Covariance (BDC) to measure the discrepancy between the logits produced by the teacher and student models. Unlike the conventional KL divergence used in traditional knowledge distillation, BDC captures not only linear relationships but also nonlinear dependencies, thereby overcoming the limitations of KL divergence and enabling the student model to learn more effectively from the teacher model. Additionally, our method aligns the discrepancies between the teacher and student models from both intra-class and inter-class perspectives. Extensive experiments demonstrate that our method achieves state-of-the-art (SOTA) performance across various network architectures and datasets. The code and resources related to this work are available at the following link: https://github.com/hengyin23654/BDCKD.