Product of Experts with LLMs: Boosting Performance on ARC Is a Matter of Perspective
Daniel Franzen, Jan Disselhoff, David Hartmann
Abstract
The Abstraction and Reasoning Corpus (ARC-AGI) poses a significant challenge for large language models (LLMs), exposing limitations in their abstract reasoning abilities. In this work, we leverage task-specific data augmentations throughout the training, generation, and scoring phases, and employ a depth-first search algorithm to generate diverse, high-probability candidate solutions. Furthermore, we utilize the LLM not only as a generator but also as a scorer, using its output probabilities to select the most promising solutions. Our method achieves a score of 71.6% (286.5/400 solved tasks) on the public ARC-AGI evaluation set, demonstrating state-of-the-art performance among publicly available approaches. While concurrent closed-source work has reported higher scores, our method distinguishes itself through its transparency, reproducibility, and remarkably low inference cost, averaging only around 2ct per task on readily available hardware.
BibTeX
@inproceedings{
franzen2025product,
title={Product of Experts with {LLM}s: Boosting Performance on {ARC} Is a Matter of Perspective},
author={Daniel Franzen and Jan Disselhoff and David Hartmann},
booktitle={Forty-second International Conference on Machine Learning},
year={2025},
url={https://openreview.net/forum?id=dsBjxI6l8W}
}