Sparse Progressive Distillation: Resolving Overfitting under Pretrain-and-Finetune Paradigm
Shaoyi Huang, Dongkuan Xu, Ian Yen, Yijue Wang, Sung-En Chang, Bingbing Li, Shiyang Chen, Mimi Xie
Abstract
Conventional wisdom in pruning Transformer-based language models is that pruning reduces the model expressiveness and thus is more likely to underfit rather than overfit. However, under the trending pretrain-and-finetune paradigm, we postulate a counter-traditional hypothesis, that is: pruning increases the risk of overfitting when performed at the fine-tuning phase. In this paper, we aim to address the overfitting problem and improve pruning performance via progressive knowledge distillation with error-bound properties. We show for the first time that reducing the risk of overfitting can help the effectiveness of pruning under the pretrain-and-finetune paradigm. Ablation studies and experiments on the GLUE benchmark show that our method outperforms the leading competitors across different tasks.
BibTeX
@inproceedings{huang-etal-2022-sparse,
title = "Sparse Progressive Distillation: Resolving Overfitting under Pretrain-and-Finetune Paradigm",
author = "Huang, Shaoyi and
Xu, Dongkuan and
Yen, Ian and
Wang, Yijue and
Chang, Sung-En and
Li, Bingbing and
Chen, Shiyang and
Xie, Mimi and
Rajasekaran, Sanguthevar and
Liu, Hang and
Ding, Caiwen",
editor = "Muresan, Smaranda and
Nakov, Preslav and
Villavicencio, Aline",
booktitle = "Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
month = may,
year = "2022",
address = "Dublin, Ireland",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2022.acl-long.16/",
doi = "10.18653/v1/2022.acl-long.16",
pages = "190--200"
}