Cross-Layer Graph Knowledge Distillation for Image Recognition
Jiaming Chu, Yanzhuo Xiang, Yuqi Li, Chuanguang Yang, Zhulin An, Yongjun Xu
Abstract
Knowledge Distillation (KD) aims to improve a light-weight student network supervised by a large teacher network. The core idea of KD is to explore valuable knowledge from the teacher. Previous works often extract information from a single sample, but ignore relation modeling among multiple samples between student and teacher. Therefore, we propose Cross-Layer Graph Knowledge Distillation (CLGKD) that conducts graph-augmented feature and relation distillation assisted by graph neural networks. We further propose a meta-learning mechanism to optimize cross-layer matching weights for promoting GKD among all student and teacher layers. Experimental results on image classification and object detection demonstrate that CLGKD achieves state-of-the-art performance compared to other KD methods. Our code is available at https://github.com/cynmzzz/ICASSP2025-CLGKD
BibTeX
@inproceedings{icassp2025_crosslayergraphk,
title = {Cross-Layer Graph Knowledge Distillation for Image Recognition},
author = {Jiaming Chu and Yanzhuo Xiang and Yuqi Li and Chuanguang Yang and Zhulin An and Yongjun Xu},
booktitle = {ICASSP 2025},
year = {2025}
}