Distributed Training with Heterogeneous Data: Bridging Median- and Mean-Based Algorithms
Recently, there is a growing interest in the study of median-based algorithms for distributed non-convex optimization. Two prominent examples include signSGD with majority vote, an effective approach for communication reduction via 1-bit compression on the local gradients, and medianSGD, an algorith…