2026
SimulCost: A Cost-Aware Benchmark and Toolkit for Automating Physics Simulations with LLMs
ICML 2026poster
Evaluating LLM agents for scientific tasks has focused on token costs while ignoring tool-use costs like simulation time and experimental resources. As a result, metrics like pass@k become impractical under realistic budget constraints. To address this gap, we introduce SimulCost, the first benchmar…