Enhancing Federated Knowledge Distillation in Heterogeneous and Non-IID Scenarios
Wenjie Lv, Yu He, Sen Liu, Xingjun Ma, Xiang Liu, Guangnan Ye, Hongfeng Chai
Abstract
Federated Learning (FL) allows multiple participants to train models together while keeping their data private. Some FL frameworks use Knowledge Distillation to address model heterogenity, but many struggle in non-IID and heterogeneous environments, making it hard for clients to learn from each other. In this work, we show that the entropy of the softmax-averaged logits from clients reflects the model’s convergence. Based on this, we propose a new loss function, Sharpened Symmetric KL Divergence Loss (SSKL), which combines KL and Reverse KL Divergence with Label Sharpening to reduce the impact of non-IID data. Experiments demonstrate that our approach improves performance and reduces accuracy decline in non-IID and heterogeneous settings.
BibTeX
@inproceedings{icassp2025_enhancingfederat,
title = {Enhancing Federated Knowledge Distillation in Heterogeneous and Non-IID Scenarios},
author = {Wenjie Lv and Yu He and Sen Liu and Xingjun Ma and Xiang Liu and Guangnan Ye and Hongfeng Chai},
booktitle = {ICASSP 2025},
year = {2025}
}