ToolSandbox: A Stateful, Conversational, Interactive Evaluation Benchmark for LLM Tool Use Capabilities
Jiarui Lu, Thomas Holleis, Yizhe Zhang, Bernhard Aumayer, Feng Nan, Haoping Bai, Shuang Ma, Shen Ma
Abstract
Recent large language models (LLMs) advancements sparked a growing research interest in tool assisted LLMs solving real-world challenges, which calls for comprehensive evaluation of tool-use capabilities. While previous works focused on either evaluating over stateless web services (RESTful API), based on a single turn user prompt, or an off-policy dialog trajectory, ToolSandbox includes stateful tool execution, implicit state dependencies between tools, a built-in user simulator supporting on-policy conversational evaluation and a dynamic evaluation strategy for intermediate and final milestones over arbitrary trajectory. We show that open source and proprietary models has a significant performance gap, and complex tasks like State Dependency, Canonicalization and Insufficient Information defined in ToolSandbox are challenging even the most capable SOTA LLMs, providing brand-new insights to tool-use LLM capabilities. Datasets and evaluation scripts of ToolSandbox are released at <placeholder>.
BibTeX
@inproceedings{lu-etal-2025-toolsandbox,
title = "{T}ool{S}andbox: A Stateful, Conversational, Interactive Evaluation Benchmark for {LLM} Tool Use Capabilities",
author = "Lu, Jiarui and
Holleis, Thomas and
Zhang, Yizhe and
Aumayer, Bernhard and
Nan, Feng and
Bai, Haoping and
Ma, Shuang and
Ma, Shen and
Li, Mengyu and
Yin, Guoli and
Wang, Zirui and
Pang, Ruoming",
editor = "Chiruzzo, Luis and
Ritter, Alan and
Wang, Lu",
booktitle = "Findings of the Association for Computational Linguistics: NAACL 2025",
month = apr,
year = "2025",
address = "Albuquerque, New Mexico",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2025.findings-naacl.65/",
pages = "1160--1183",
ISBN = "979-8-89176-195-7"
}