ITBench: Evaluating AI Agents across Diverse Real-World IT Automation Tasks
Saurabh Jha, Rohan R. Arora, Yuji Watanabe, Takumi Yanagawa, Yinfang Chen, Jackson Clark, Bhavya Bhavya, Mudit Verma
Abstract
Realizing the vision of using AI agents to automate critical IT tasks depends on the ability to measure and understand effectiveness of proposed solutions. We introduce ITBench, a framework that offers a systematic methodology for benchmarking AI agents to address real-world IT automation tasks. Our initial release targets three key areas: Site Reliability Engineering (SRE), Compliance and Security Operations (CISO), and Financial Operations (FinOps). The design enables AI researchers to understand the challenges and opportunities of AI agents for IT automation with push-button workflows and interpretable metrics. IT-Bench includes an initial set of 102 real-world scenarios, which can be easily extended by community contributions. Our results show that agents powered by state-of-the-art models resolve only 11.4% of SRE scenarios, 25.2% of CISO scenarios, and 25.8% of FinOps scenarios (excluding anomaly detection). For FinOps-specific anomaly detection (AD) scenarios, AI agents achieve an F1 score of 0.35. We expect ITBench to be a key enabler of AI-driven IT automation that is correct, safe, and fast. IT-Bench, along with a leaderboard and sample agent implementations, is available at https://github.com/ibm/itbench.
BibTeX
@inproceedings{
jha2025itbench,
title={{ITB}ench: Evaluating {AI} Agents across Diverse Real-World {IT} Automation Tasks},
author={Saurabh Jha and Rohan R. Arora and Yuji Watanabe and Takumi Yanagawa and Yinfang Chen and Jackson Clark and Bhavya Bhavya and Mudit Verma and Harshit Kumar and Hirokuni Kitahara and Noah Zheutlin and Saki Takano and Divya Pathak and Felix George and Xinbo Wu and Bekir O Turkkan and Gerard Vanloo and Michael Nidd and Ting Dai and Oishik Chatterjee and Pranjal Gupta and Suranjana Samanta and Pooja Aggarwal and Rong Lee and Jae-wook Ahn and Debanjana Kar and Amit Paradkar and Yu Deng and Pratibha Moogi and Prateeti Mohapatra and Naoki Abe and Chandrasekhar Narayanaswami and Tianyin Xu and Lav R. Varshney and Ruchi Mahindru and Anca Sailer and Laura Shwartz and Daby Sow and Nicholas C. M. Fuller and Ruchir Puri},
booktitle={Forty-second International Conference on Machine Learning},
year={2025},
url={https://openreview.net/forum?id=jP59rz1bZk}
}