MERGE$^3$: Efficient Evolutionary Merging on Consumer-grade GPUs
Tommaso Mencattini, Robert Adrian Minut, Donato Crisostomi, Andrea Santilli, Emanuele Rodolà
Abstract
Evolutionary model merging enables the creation of high-performing multi-task models but remains computationally prohibitive for consumer hardware. We introduce MERGE$^3$, an efficient framework that makes evolutionary merging of Large Language Models (LLMs) feasible on a single GPU by reducing fitness computation costs 50× while retaining a large fraction of the original performance. MERGE$^3$ achieves this by **E**xtracting a reduced dataset for evaluation, **E**stimating model abilities using Item Response Theory (IRT), and **E**volving optimal merges via IRT-based performance estimators. Our method enables state-of-the-art multilingual and cross-lingual merging, transferring knowledge across languages with significantly lower computational overhead. We provide theoretical guarantees and an open-source library, democratizing high-quality model merging.
BibTeX
@inproceedings{
mencattini2025merge,
title={{MERGE}\${\textasciicircum}3\$: Efficient Evolutionary Merging on Consumer-grade {GPU}s},
author={Tommaso Mencattini and Robert Adrian Minut and Donato Crisostomi and Andrea Santilli and Emanuele Rodol{\`a}},
booktitle={Forty-second International Conference on Machine Learning},
year={2025},
url={https://openreview.net/forum?id=qFXDv0X4yc}
}