Limin Lu

Row-stochastic matrices can provably outperform doubly stochastic matrices in decentralized learning

Decentralized learning often involves a weighted global loss with heterogeneous node weights $\lambda$. We revisit two natural strategies for incorporating these weights: (i) embedding them into the local losses to retain a uniform weight (and thus a doubly stochastic matrix), and (ii) keeping the o…