Beyond Structural Symmetries: Linear Mode Connectivity via Neuron Identifiability
Vincent Bürgin, Daniel Herbst, Ya-Wei Eileen Lin, Stefanie Jegelka
Abstract
Many striking phenomena in deep learning, such as linear mode connectivity and the structured behavior of training dynamics, are closely tied to parameter symmetries: transformations that leave the realized function unchanged. Despite growing attention to structural parameter symmetries, the exact interplay between parameters, data, and representations remains underexplored. To investigate this, we develop a theoretical framework of effective function classes defined by the neurons' induced functions restricted to the representation subspace. We then formalize *effective symmetry breaking* via neuron identifiability across independent training runs. Our analysis shows that neural networks can admit large families of approximately equivalent solutions even in *structurally asymmetric* models. This allows us to disentangle the effects of data-specific and architectural symmetries. We further show that neuron identifiability enables representation merging *without prior alignment*, and characterize when such merging admits a linear low-loss connecting path. These findings highlight the role of effective function classes in affecting the loss landscape.
BibTeX
@inproceedings{
burgin2026beyond,
title={Beyond Structural Symmetries: Linear Mode Connectivity via Neuron Identifiability},
author={Vincent B{\"u}rgin and Daniel Herbst and Ya-Wei Eileen Lin and Stefanie Jegelka},
booktitle={Forty-third International Conference on Machine Learning},
year={2026},
url={https://openreview.net/forum?id=FJLz8swPpd}
}