2024
The Role of Masking for Efficient Supervised Knowledge Distillation of Vision Transformers
ECCV 2024poster
"Knowledge distillation is an effective method for training lightweight vision models. However, acquiring teacher supervision for training samples is often costly, especially from large-scale models like vision transformers (ViTs). In this paper, we develop a simple framework to reduce the supervisi…