Second-Order Wireless Federated Leaning via Nonparametric Hessian Estimation

Abstract

Quasi-Newton algorithms estimate the second-order information of loss landscape from its first-order information. They hence propose a promising solution for communication-efficient federated learning (FL), as they reduce the required number of training rounds while avoiding the necessity of exchanging local Hessians over the network. Despite that, the quasi-Newton approaches prove less effective in wireless FL, as the noisy aggregation in this case causes bias in the estimate of the Newton direction. This paper proposes a novel second-order wireless FL algorithm. The pivotal innovation lies in the server’s ability to estimate the global Hessian based on a window of noisy aggregations. The server acquires this ability by computing a stochastic estimator of the global Hessian under a Gaussian prior belief. Numerical experiments show that the proposed scheme can compute a less-biased estimator of the Newton direction, and hence a superior learning performance, as compared to the baseline.