CodeBPE: Investigating Subtokenization Options for Large Language Model Pretraining on Source Code

Nadezhda Chirkova; Sergey Troshin

ICLR 2023poster7 citations

CodeBPE: Investigating Subtokenization Options for Large Language Model Pretraining on Source Code

Abstract

Recent works have widely adopted large language model pretraining for source code, suggested source code-specific pretraining objectives and investigated the applicability of various Transformer-based language model architectures for source code. This work investigates another important aspect of such models, the effect of different subtokenization options, and aims at identifying most effective and length-efficient subtokenizations, taking into account source code specifics. We propose subtokenziation that reduces average length by 17--40% without downstream performance drop, and show that a carefully chosen subtokenization may improve quality by 0.5-2%, possibly with some length increase.

source code processingtokenizationbyte-pair encoding

BibTeX

@inproceedings{
chirkova2023codebpe,
title={Code{BPE}: Investigating Subtokenization Options for Large Language Model Pretraining on Source Code},
author={Nadezhda Chirkova and Sergey Troshin},
booktitle={The Eleventh International Conference on Learning Representations },
year={2023},
url={https://openreview.net/forum?id=htL4UZ344nF}
}