DOBF: A Deobfuscation Pre-Training Objective for Programming Languages
Marie-anne Lachaux, Baptiste Roziere, Marc Szafraniec, Guillaume Lample
Abstract
Recent advances in self-supervised learning have dramatically improved the state of the art on a wide variety of tasks. However, research in language model pre-training has mostly focused on natural languages, and it is unclear whether models like BERT and its variants provide the best pre-training when applied to other modalities, such as source code. In this paper, we introduce a new pre-training objective, DOBF, that leverages the structural aspect of programming languages and pre-trains a model to recover the original version of obfuscated source code. We show that models pre-trained with DOBF significantly outperform existing approaches on multiple downstream tasks, providing relative improvements of up to 12.2% in unsupervised code translation, and 5.3% in natural language code search. Incidentally, we found that our pre-trained model is able to deobfuscate fully obfuscated source files, and to suggest descriptive variable names.
BibTeX
@inproceedings{
lachaux2021dobf,
title={{DOBF}: A Deobfuscation Pre-Training Objective for Programming Languages},
author={Marie-anne Lachaux and Baptiste Roziere and Marc Szafraniec and Guillaume Lample},
booktitle={Advances in Neural Information Processing Systems},
editor={A. Beygelzimer and Y. Dauphin and P. Liang and J. Wortman Vaughan},
year={2021},
url={https://openreview.net/forum?id=3ez9BSHTNT}
}