Generalization Guarantee of Decentralized Learning with Heterogeneous Data

Abstract

Decentralized learning, which facilitates joint model training across geographically scattered devices, has gained significant attention in the field of signal and information processing in recent years. While the optimization errors of decentralized learning algorithms have been extensively studied, their generalization errors remain relatively under-explored. As the generalization errors reflect the scalability of the trained models on unseen data and are crucial in determining the performance of the trained models in real-world applications, understanding the generalization errors of decentralized learning algorithms is of paramount importance. In this paper, we present the first fine-grained generalization error analysis for decentralized learning with heterogeneous data as well as under mild assumptions, in contrast to prior studies that consider the homogeneous data and/or rely on a stringent bounded stochastic gradient assumption. Our results shed light on the impact of data heterogeneity, model initialization and stochastic gradient noise – factors that have not been previously investigated – on the generalization error of decentralized learning. Numerical experiments are conducted to validate our theoretical findings.