SWE-Compass: Towards Unified Evaluation of Agentic Coding Abilities for Large Language Models
Jingxuan Xu, Ken Deng, Weihao Li, Songwei Yu, Haoyang Huang, Yifan Yao, Huaixi Tang, Zhiyi Lai
Abstract
Evaluating large language models (LLMs) for software engineering has been limited by narrow task coverage, language bias, and insufficient alignment with real-world developer workflows. Existing benchmarks often focus on algorithmic problems or Python-centric bug fixing, leaving critical dimensions of software engineering underexplored. To address these gaps, we introduce SWE-Compass, a comprehensive benchmark that unifies heterogeneous code-related evaluations into a structured and production-aligned framework. SWE-Compass spans 8 task types, 8 programming scenarios, and 10 programming languages, with 2000 high-quality instances curated from authentic GitHub pull requests and refined through systematic filtering and validation. We benchmark ten state-of-the-art LLMs under two agentic frameworks, SWE-Agent and Claude Code, revealing a clear hierarchy of difficulty across task types, languages, and scenarios. Moreover, by aligning evaluation with real-world developer practices, we hope SWE-Compass can provide a rigorous and reproducible foundation for diagnosing and advancing agentic coding capabilities in large language models.
BibTeX
@inproceedings{
xu2026swecompass,
title={{SWE}-Compass: Towards Unified Evaluation of Agentic Coding Abilities for Large Language Models},
author={Jingxuan Xu and Ken Deng and Weihao Li and Songwei Yu and Haoyang Huang and Xinping Lei and Yifan Yao and Huaixi Tang and Zhiyi Lai and Kepeng Lei and Zizheng Zhan and Yanan Wu and Chenchen Zhang and Wenqiang Zhu and Wen Xiang and Zongxian Feng and Han Li and Junqi Xiong and Dailin Li and Zuchen Gao and Kun Wu and Yuanxing Zhang and Wuxuan Gong and Ziyuan Gao and Guanxiang Wang and Yirong Xue and Mengfei Xie and Xiaojiang Zhang and Jinghui Wang and Wenhao Zhuang and Zheng Lin and Huiming Wang and Zhaoxiang Zhang and Yuqun Zhang and Haotian Zhang and Ming Sun and Chen Bin and Jiaheng Liu},
booktitle={Forty-third International Conference on Machine Learning},
year={2026},
url={https://openreview.net/forum?id=MHXIvy3bsb}
}