AlgoTune: Can Language Models Speed Up General-Purpose Numerical Programs?
Ori Press, Brandon Amos, Haoyu Zhao, Yikai Wu, Samuel Ainsworth, Dominik Krupke, Patrick Kidger, Touqir Sajed
Abstract
Despite progress in language model (LM) capabilities, evaluations have thus far focused on models' performance on tasks that humans have previously solved, including in programming (SWE-Bench) and mathematics (FrontierMath). We therefore propose testing models' ability to design and implement algorithms in an open-ended benchmark: We task LMs with writing code that efficiently solves computationally challenging problems in computer science, physics, and mathematics. Our AlgoTune benchmark consists of 120 tasks collected from domain experts and a framework for validating and timing LM-synthesized solution code, which is compared to reference implementations from popular open-source packages. In addition, we develop a baseline LM agent, AlgoTuner, and evaluate its performance across a suite of frontier models. AlgoTuner achieves an average 1.58x speedup against reference solvers, including methods from packages such as SciPy, scikit-learn and CVXPY. However, we find that current models fail to discover algorithmic innovations, instead preferring surface-level optimizations. We hope that AlgoTune catalyzes the development of LM agents exhibiting creative problem solving beyond state-of-the-art human performance.
BibTeX
@inproceedings{
press2025algotune,
title={AlgoTune: Can Language Models Speed Up General-Purpose Numerical Programs?},
author={Ori Press and Brandon Amos and Haoyu Zhao and Yikai Wu and Samuel Ainsworth and Dominik Krupke and Patrick Kidger and Touqir Sajed and Bartolomeo Stellato and Jisun Park and Nathanael Bosch and Eli Meril and Albert Steppi and Arman Zharmagambetov and Fangzhao Zhang and David P{\'e}rez-Pi{\~n}eiro and Alberto Mercurio and Ni Zhan and Talor Abramovich and Kilian Lieret and Hanlin Zhang and Shirley Huang and Matthias Bethge and Ofir Press},
booktitle={The Thirty-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track},
year={2025},
url={https://openreview.net/forum?id=dF1tD9hjvn}
}