Which LLM Multi-Agent Protocol to Choose?
Hongyi Du, Jiaqi Su, Jisen Li, Lijie Ding, Yingxuan Yang, Peixuan Han, Robert Tang, Kunlun Zhu
Abstract
As large-scale multi-agent systems evolve, the communication protocol layer has become a critical, yet understudied, component affecting system performance and reliability. Despite a range of protocols, such as JSON-RPC, A2A, ANP, and ACP, protocol selection remains ad hoc. To address this, we introduce ProtocolBench, a benchmark designed to evaluate agent communication protocols across task utility, communication overhead, system performance, and resilience under failure. ProtocolBench uses a three-layer architecture with protocol adapters for fair com- parison, diverse scenarios (e.g., document aggregation, collaborative coding), and detailed telemetry. Our results show protocol choice can impact task completion time by up to 36%, communication overhead by 3.5 seconds, and resilience with statistically observable differences. We also propose ProtocolRouter, a learnable protocol routing system that dynamically selects protocols based on runtime con- ditions, improving performance by up to 18% compared to individual protocols. Our findings highlight that hybrid protocol deployments outperform homogeneous ones by approximately 6.6%, with negligible protocol translation overhead. We release ProtocolBench as an open-source framework to standardize protocol evaluation and improve multi-agent system reliability at scale.
BibTeX
@inproceedings{
du2026protocolbench,
title={ProtocolBench: Which {LLM} MultiAgent Protocol to Choose?},
author={Hongyi Du and Jiaqi Su and Jisen Li and Lijie Ding and Yingxuan Yang and Peixuan Han and Xiangru Tang and Kunlun Zhu and Jiaxuan You},
booktitle={Forty-third International Conference on Machine Learning},
year={2026},
url={https://openreview.net/forum?id=GZRiXE2zBe}
}