Learning Dynamics of Low-Precision Clipped SGD with Momentum

Abstract

In this work, we present and study a low-precision variant of the stochastic gradient descent (SGD) algorithm with adaptive quantization. In particular, fixed-rate probabilistic uniform quantizers with varying quantization steps and mid-values are used to compress the parameter vectors. Gradient clipping and momentum are used to guarantee that the quantizer inputs fall within the representable region of the fixed-rate quantizer and to reduce the impact of the stochastic gradient noise, respectively. We show that, despite the low-precision representation, the quantized variant of the clipped SGD algorithm with momentum is able to converge in the mean-square-error sense. Simulation results illustrate the theoretical findings and the effectiveness of the proposed approach.