Better Language Models of Code through Self-Improvement
Hung To, Nghi Bui, Jin L.C. Guo, Tien Nguyen
Abstract
Pre-trained language models for code (PLMCs) have gained attention in recent research. These models are pre-trained on large-scale datasets using multi-modal objectives. However, fine-tuning them requires extensive supervision and is limited by the size of the dataset provided. We aim to improve this issue by proposing a data augmentation framework using knowledge distillation. Our framework utilizes knowledge gained during the pre-training and fine-tuning stage to augment training data, which is then used for the next step. We incorporate this framework into the state-of-the-art language models, such as CodeT5, CodeBERT, and UnixCoder. The results show that our framework significantly improves PLMCs’ performance in sequence-generation tasks, such as code summarization and code generation in the CodeXGLUE benchmark.
BibTeX
@inproceedings{to-etal-2023-better,
title = "Better Language Models of Code through Self-Improvement",
author = "To, Hung and
Bui, Nghi and
Guo, Jin L.C. and
Nguyen, Tien",
editor = "Rogers, Anna and
Boyd-Graber, Jordan and
Okazaki, Naoaki",
booktitle = "Findings of the Association for Computational Linguistics: ACL 2023",
month = jul,
year = "2023",
address = "Toronto, Canada",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2023.findings-acl.823/",
doi = "10.18653/v1/2023.findings-acl.823",
pages = "12994--13002"
}