Khanh Vinh Bui

On Linear Mode Connectivity of Mixture-of-Experts Architectures

Linear Mode Connectivity (LMC) is a notable phenomenon in the loss landscapes of neural networks, wherein independently trained models have been observed to be connected—up to permutation symmetries—by linear paths in parameter space along which the loss remains consistently low. This observation ch…