Sunjae Jeong

SiNGER: A Clearer Voice Distills Vision Transformers Further

Vision Transformers are widely adopted as the backbone of vision foundation models, but they are known to produce high-norm artifacts that degrade representation quality. When knowledge distillation transfers these features to students, high-norm artifacts dominate the objective, so students overfit…