Self Knowledge Distillation Based On Layer-Wise Weighted Feature Imitation For Efficient Object Detection

Abstract

Knowledge Distillation (KD)[1] is a widely-used technology to inherit information from cumbersome teacher models to compact student models, consequently realizing model compression and acceleration. Compared with image classification, object detection is a more complex task, and designing specific KD methods for object detection is non-trivial. In this paper, we propose Layer-wise Weighted Feature Imitation (LWFI), the model loaded with pre-trained parameters acts as a teacher to guide students who have not been pretrained. We use the feature maps of multiple intermediate positions in the teacher network to guide the corresponding positions in the student network, and allocate corresponding weights according to the magnitude of distillation loss. We have carried out a series of experiments on the VOC and KITTI datasets. Specifically, YOLOv6N on VOC improved from 66.0% to 67.3%, YOLOv6S improved from 70.8% to 71.9%, and on KITTI, YOLOv6N increased from 56.4% to 58.1%, outperforming similar-sized networks.