EMNLP 2024finding21 citations
On Leakage of Code Generation Evaluation Datasets
Alexandre Matton, Tom Sherborne, Dennis Aumiller, Elena Tommasone, Milad Alizadeh, Jingyi He, Raymond Ma, Maxime Voisin
Abstract
In this paper, we consider contamination by code generation test sets, in particular in their use in modern large language models.We discuss three possible sources of such contamination and show findings supporting each of them: (i) direct data leakage, (ii) indirect data leakage through the use of synthetic data and (iii) overfitting to evaluation sets during model selection.To address this, we release Less Basic Python Problems (LBPP): an uncontaminated new benchmark of 161 prompts with their associated Python solutions. LBPP is released at https://huggingface.co/datasets/CohereForAI/lbpp
BibTeX
@inproceedings{matton-etal-2024-leakage,
title = "On Leakage of Code Generation Evaluation Datasets",
author = "Matton, Alexandre and
Sherborne, Tom and
Aumiller, Dennis and
Tommasone, Elena and
Alizadeh, Milad and
He, Jingyi and
Ma, Raymond and
Voisin, Maxime and
Gilsenan-McMahon, Ellen and
Gall{\'e}, Matthias",
editor = "Al-Onaizan, Yaser and
Bansal, Mohit and
Chen, Yun-Nung",
booktitle = "Findings of the Association for Computational Linguistics: EMNLP 2024",
month = nov,
year = "2024",
address = "Miami, Florida, USA",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2024.findings-emnlp.772/",
doi = "10.18653/v1/2024.findings-emnlp.772",
pages = "13215--13223"
}