Zhaodong Bing

Efficient One Pass Self-Distillation with Zipf’s Label Smoothing

"Self-distillation exploits non-uniform soft supervision from itself during training and improves performance without any runtime cost. However, the overhead during training is often overlooked, and yet reducing time and memory overhead during training is increasingly important in the giant models’…