Gradient-Oriented Clustered Federated Learning With Efficient Knowledge Sharing in Non-IID Settings

Abstract

We present a novel Clustered Federated Learning (CFL) approach that efficiently shares knowledge among clusters to address non-Independent and Identically Distributed (non-IID) settings. Although conventional CFL has demonstrated strong performance in non-IID settings, a fundamental challenge, the lack of knowledge sharing among clusters remains as a problem to be solved for improvement of generalization. To overcome this limitation, we utilize personalized layers for specific tasks while retaining globally shared layers after clustering. By separating the layers of the model according to their functions, our method enables the model to be well personalized without sacrificing generalizability. Furthermore, we cluster clients based on the gradients of specific layers. Our efficient CFL method can better characterize the data of each client, reduce the memory demand of the client, and mitigate the computational burden of the FL server. We evaluate the effectiveness of our method in various data distribution settings.