Prototypical Fine-Tuning: Towards Robust Performance under Varying Data Sizes
Yiqiao Jin, Xiting Wang, Yaru Hao, Yizhou Sun, Xing Xie
Abstract
In this paper, we move towards combining large parametric models with non-parametric prototypical networks. We propose prototypical fine-tuning, a novel prototypical framework for fine-tuning pretrained language models (LM), which automatically learns a bias to improve predictive performance for varying data sizes, especially low-resource settings. Our prototypical fine-tuning approach can automatically adjust the model capacity according to the number of data points and the model's inherent attributes. Moreover, we propose four principles for effective prototype fine-tuning towards the optimal solution. Experimental results across various datasets show that our work achieves significant performance improvements under various low-resource settings, as well as comparable and usually better performances in high-resource scenarios.
BibTeX
@article{Jin_Wang_Hao_Sun_Xie_2023, title={Prototypical Fine-Tuning: Towards Robust Performance under Varying Data Sizes}, volume={37}, url={https://ojs.aaai.org/index.php/AAAI/article/view/26524}, DOI={10.1609/aaai.v37i11.26524}, abstractNote={In this paper, we move towards combining large parametric models with non-parametric prototypical networks. We propose prototypical fine-tuning, a novel prototypical framework for fine-tuning pretrained language models (LM), which automatically learns a bias to improve predictive performance for varying data sizes, especially low-resource settings. Our prototypical fine-tuning approach can automatically adjust the model capacity according to the number of data points and the model’s inherent attributes. Moreover, we propose four principles for effective prototype fine-tuning towards the optimal solution. Experimental results across various datasets show that our work achieves significant performance improvements under various low-resource settings, as well as comparable and usually better performances in high-resource scenarios.}, number={11}, journal={Proceedings of the AAAI Conference on Artificial Intelligence}, author={Jin, Yiqiao and Wang, Xiting and Hao, Yaru and Sun, Yizhou and Xie, Xing}, year={2023}, month={Jun.}, pages={12968-12976} }