Functional Equivalence in Attention: A Comprehensive Study with Applications to Linear Mode Connectivity
Neural network parameter spaces are inherently non-injective, as distinct parameter configurations can realize identical functions through functional equivalence. While this symmetry is well understood in classical fully connected and convolutional models, it becomes substantially more intricate in …