General Dynamic Regularization Federated Learning with Hybrid Sharpness-Aware Minimization

Abstract

One of the main challenges in federated learning is its non-independent and identically distributed (non-IID) nature, where independent client training leads to overfitting and model deviations, negatively impacting overall performance. To address this, most research focuses on aligning local and global models to reduce client drift. However, existing algorithms use Empirical Risk Minimization (ERM) as the local optimizer, leading the global model to sharp minima, increasing bias in some clients and lowering generalization. To address these challenges, we propose GFed-HSAM, a general federated learning method that improves both local and global model generalization. GFed-HSAM uses Hybrid Sharpness-Aware Minimization (HSAM) as the local optimizer to smooth gradients with zeroth-order and first-order sharpness measures. It also includes a dynamic regularizer (DR) to align global and local models at the parameter level. Experiments show that GFed-HSAM outperforms state-of-theart methods in accuracy and generalization across different data heterogeneity settings on CIFAR10/CIFAR100.