RW-KD: Sample-wise Loss Terms Re-Weighting for Knowledge Distillation
Peng Lu, Abbas Ghaddar, Ahmad Rashid, Mehdi Rezagholizadeh, Ali Ghodsi, Philippe Langlais
Abstract
Knowledge Distillation (KD) is extensively used in Natural Language Processing to compress the pre-training and task-specific fine-tuning phases of large neural language models. A student model is trained to minimize a convex combination of the prediction loss over the labels and another over the teacher output. However, most existing works either fix the interpolating weight between the two losses apriori or vary the weight using heuristics. In this work, we propose a novel sample-wise loss weighting method, RW-KD. A meta-learner, simultaneously trained with the student, adaptively re-weights the two losses for each sample. We demonstrate, on 7 datasets of the GLUE benchmark, that RW-KD outperforms other loss re-weighting methods for KD.
BibTeX
@inproceedings{lu-etal-2021-rw-kd,
title = "{RW}-{KD}: Sample-wise Loss Terms Re-Weighting for Knowledge Distillation",
author = "Lu, Peng and
Ghaddar, Abbas and
Rashid, Ahmad and
Rezagholizadeh, Mehdi and
Ghodsi, Ali and
Langlais, Philippe",
editor = "Moens, Marie-Francine and
Huang, Xuanjing and
Specia, Lucia and
Yih, Scott Wen-tau",
booktitle = "Findings of the Association for Computational Linguistics: EMNLP 2021",
month = nov,
year = "2021",
address = "Punta Cana, Dominican Republic",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2021.findings-emnlp.270/",
doi = "10.18653/v1/2021.findings-emnlp.270",
pages = "3145--3152"
}