Xinmiao Dai

Distill Vision Transformers to CNNs via Teacher Collaboration

The vision transformer (ViT) has recently emerged as a leading approach in various domains, outperforming other methods. Therefore, it is logical to explore the possibility of transferring the superior knowledge from ViT to more compact and cost-effective convolutional neural networks (CNNs). Howeve…