DependEval: Benchmarking LLMs for Repository Dependency Understanding
Junjia Du, Yadi Liu, Hongcheng Guo, Jiawei Wang, Haojian Huang, Yunyi Ni, Zhoujun Li
Abstract
While large language models (LLMs) have shown considerable promise in code generation, real-world software development demands advanced repository-level reasoning. This includes understanding dependencies, project structures, and managing multi-file changes. However, the ability of LLMs to effectively comprehend and handle complex code repositories has yet to be fully explored. To address these challenges, we introduce a hierarchical benchmark designed to evaluate repository dependency understanding(DependEval) for LLMs. The benchmark is based on 2683 repositories collected from real-world websites. It evaluates models on three core tasks: Dependency Recognition, Repository Construction, and Multi-file Editing, across 8 programming languages from actual code repositories. Our evaluation of over 25 LLMs reveals substantial performance gaps and provides valuable insights into repository-level code understanding.
BibTeX
@inproceedings{du-etal-2025-dependeval,
title = "{D}epend{E}val: Benchmarking {LLM}s for Repository Dependency Understanding",
author = "Du, Junjia and
Liu, Yadi and
Guo, Hongcheng and
Wang, Jiawei and
Huang, Haojian and
Ni, Yunyi and
Li, Zhoujun",
editor = "Che, Wanxiang and
Nabende, Joyce and
Shutova, Ekaterina and
Pilehvar, Mohammad Taher",
booktitle = "Findings of the Association for Computational Linguistics: ACL 2025",
month = jul,
year = "2025",
address = "Vienna, Austria",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2025.findings-acl.373/",
doi = "10.18653/v1/2025.findings-acl.373",
pages = "7150--7179",
ISBN = "979-8-89176-256-5"
}