Keyao Zhan

Analyzing the Role of Permutation Invariance in Linear Mode Connectivity

It was empirically observed in Enterazi et al. (2021) that when accounting for the permutation invariance of neural networks, there is likely no loss barrier along the linear interpolation between two SGD solutions -- a phenomenon known as linear mode connectivity (LMC) modulo permutation. This phen…