MCPMark: A Benchmark for Stress-Testing Realistic and Comprehensive MCP Use

Abstract

The MCP standardizes how LLMs interact with external systems, forming the foundation for general agents. However, existing MCP benchmarks remain narrow in scope: they focus on read-heavy tasks or tasks with limited interaction depth, and fail to capture the complexity and realism of real-world workflows. To address this, we propose \texttt{MCPMark}, a benchmark designed to evaluate realistic and comprehensive MCP use, comprising $127$ high-quality tasks collaboratively created by human experts and AI agents. Specifically, each task starts from a curated initial state and incldes a programmatic script for automatic verification. Moreover, these tasks require richer and more varied interactions with the environment, involving diverse create, read, update, and delete (CRUD) operations. We conduct comprehensive evaluation of cutting-edge LLMs using a minimal agent framework that operates in a tool-calling loop. Empirical results show that the best-performing model, \texttt{gpt-5-medium}, reaches only $52.56$\% pass@1 and $33.86$\% pass^4, while other widely regarded strong models, including \texttt{claude-sonnet-4} and \texttt{o3}, fall below $30$\% pass@1 and $15$\% pass^4. On average, LLMs require $16.18$ execution turns and $17.38$ tool calls per task, substantially exceeding those in previous MCP benchmarks and demonstrating the stress-testing nature of \texttt{MCPMark}.