Suk-Jin Hong

Understanding and Improving Knowledge Distillation for Quantization Aware Training of Large Transformer Encoders

Knowledge distillation (KD) has been a ubiquitous method for model compression to strengthen the capability of a lightweight model with the transferred knowledge from the teacher. In particular, KD has been employed in quantization-aware training (QAT) of Transformer encoders like BERT to improve th…