Jegwang Ryu

The Role of Masking for Efficient Supervised Knowledge Distillation of Vision Transformers

"Knowledge distillation is an effective method for training lightweight vision models. However, acquiring teacher supervision for training samples is often costly, especially from large-scale models like vision transformers (ViTs). In this paper, we develop a simple framework to reduce the supervisi…