2026
Extra-Merge: Tracing the Rank-1 Subspace of Model Merging in Language Model Pre-Training
ICML 2026poster
Model merging has emerged as a lightweight paradigm for enhancing Large Language Models (LLMs), yet its underlying mechanisms remain poorly understood. In this work, we analyze late-stage pre-training trajectories and uncover a \textbf{Rank-1 Subspace} phenomenon: while raw optimization steps oscill…