Post-Hoc Merging is Not Enough: Many-Shot Model Merging with Loss-Gap Balancing

Abstract

Model merging has become a practical post-training strategy for building a single multi-task large language model (LLM) by combining multiple task-specialized models, avoiding costly joint training. However, most existing approaches rely on post-hoc merging, in which task-specific models are merged only once after training. This one-shot aggregation often suffers from task interference, leading to *information erasure* across individual tasks. In this work, we show that replacing post-hoc merging with an iterative *many-shot merging* protocol is effective in improving multi-task performance. Building on this insight, we propose **METIS**, **M**itigating **E**rasure from **T**ask **I**nterference for **S**table many-shot merging. METIS is a loss-aware many-shot merging method that stabilizes iterative integration through task-wise loss-gap weighting and consensus-based masking. Notably, METIS exhibits significant performance improvement on the worst-performing task, effectively mitigating information erasure.