Improving Knowledge Distillation via Cross-Modal Insights from CLIP

Abstract

Knowledge distillation enhances student performance by transferring knowledge from teachers. However, existing approaches often suffer from a lack of knowledge diversity. To address this issue, we propose Cross-Modal Insights Knowledge Distillation (CIKD), a novel framework that leverages the cross-modal knowledge provided by the CLIP to improve student performance. Our approach constructs outlier factors to identify and manage outlier samples, enabling dynamic adjustment of distillation intensity. We then combine the text features from CLIP with the intermediate layer features of the student to generate new logit outputs through dot product operations, guided by the logit output of the CLIP. Experimental results demonstrate that CIKD optimizes the distillation process and effectively utilizes the multi-modal information from CLIP, resulting in enhanced knowledge transfer. Extensive experiments on the CIFAR-100 and Tiny-ImageNet datasets validate the superior performance of CIKD. The code is available at https://github.com/JT-Zhou/CIKD.