DI-BENCH: Benchmarking Large Language Models on Dependency Inference with Testable Repositories at Scale
Linghao Zhang, Junhao Wang, Shilin He, Chaoyun Zhang, Yu Kang, Bowen Li, Jiaheng Wen, Chengxing Xie
Abstract
Large Language Models have advanced automated software development, however, it remains a challenge to correctly infer dependencies, namely, identifying the internal components and external packages required for a repository to successfully run. Existing studies highlight that dependency-related issues cause over 40% of observed runtime errors on the generated repository. To address this, we introduce DI-BENCH, a large-scale benchmark and evaluation framework specifically designed to assess LLMs’ capability on dependency inference. The benchmark features 581 repositories with testing environments across Python, C#, Rust, and JavaScript. Extensive experiments with textual and execution-based metrics reveal that the current best-performing model achieves only a 48% execution pass rate on Python, indicating significant room for improvement. DI-BENCH establishes a new viewpoint for evaluating LLM performance on repositories, paving the way for more robust end-to-end software synthesis.
BibTeX
@inproceedings{zhang-etal-2025-di,
title = "{DI}-{BENCH}: Benchmarking Large Language Models on Dependency Inference with Testable Repositories at Scale",
author = "Zhang, Linghao and
Wang, Junhao and
He, Shilin and
Zhang, Chaoyun and
Kang, Yu and
Li, Bowen and
Wen, Jiaheng and
Xie, Chengxing and
Wang, Maoquan and
Huang, Yufan and
Nallipogu, Elsie and
Lin, Qingwei and
Dang, Yingnong and
Rajmohan, Saravan and
Zhang, Dongmei and
Zhang, Qi",
editor = "Che, Wanxiang and
Nabende, Joyce and
Shutova, Ekaterina and
Pilehvar, Mohammad Taher",
booktitle = "Findings of the Association for Computational Linguistics: ACL 2025",
month = jul,
year = "2025",
address = "Vienna, Austria",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2025.findings-acl.528/",
doi = "10.18653/v1/2025.findings-acl.528",
pages = "10134--10153",
ISBN = "979-8-89176-256-5"
}