Differentially Private and Communication-efficient Decentralized Learning Using Deep Quantizers

Abstract

Decentralized learning has emerged as a popular method due to its excellent scalability and parallel implementation of stochastic gradient methods. However, the main challenges in decentralized learning include the communication overhead and privacy concerns associated with sharing gradients with neighbors. In this work, we design a novel deep quantizer to address these simultaneously, ensuring differential privacy and communication efficiency. Specifically, we learn a deep quantizer such that the induced quantization noise can be modeled as additive Gaussian noise. We also propose an encoding-decoding scheme that ensures unbiasedness and bounded variance for the quantized output. For smooth objective functions, we bound the function values in terms of the norm of the gradient, the second-order gradient moment, and the induced quantization noise. We demonstrate superior performance of decentralized stochastic gradient with the proposed deep quantizer for a least square regression problem. Additionally, we analyze the performance of the proposed approach for various quantization levels, privacy budgets, batch sizes, and numbers of nodes in the network.