Explicit Spatial Hint and Implicit Logits Relation: Distilling Heterogeneous Knowledge From Vision Transformer to CNN

Abstract

A lightweight Convolutional Neural Network (CNN) typically requires knowledge transfer from a large powerful network before it is employed in resource-limited edge devices. Vision Transformer (ViT) possesses an unparalleled capability for global modeling but remains largely unexplored in Knowledge Distillation (KD). The main reason is that the gap in receptive fields between ViT and CNN causes representation discrepancy and logits confidence-bias. In this paper, we propose a novel heterogeneous distillation method based on explicit spatial hint and implicit logits relation for transferring knowledge from ViT to CNN. By exploiting class discriminative regions of an input image, class attention transfer is developed to adaptively identify common regions of interest as a spatial hint, bridging the representation gap between the heterogeneous architectures. Meanwhile, a learnable projector equipped with our soft maximum function is introduced to refine the logits relation to evaluate all classification results more evenly with smoother gradient flows, which helps CNN to effectively learn with larger capacity gaps. Extensive experiments demonstrate that our proposed method can achieves state-of-the-art performance on multiple benchmark datasets, which is simple yet efficient without relying on any auxiliary network of homogeneous architectures.