Uniform Distribution Based Learnable Quantization via Self-Knowledge Distillation

Abstract

Quantization is an effective method for compressing DNNs with enormous parameters and improving inference efficiency especially on resource-constrained devices like mobile phones. However, it is difficult to quantize models to extremely low-bit while maintaining considerable accuracy compared to their full-precision counterparts, because the information representation capacity of quantized models declines drastically as bit-widths reducing. To alleviate this problem, we propose an improved learnable quantizer modified from LCQ and adopt a self-knowledge distillation framework to assist quantization. Our method parameterizes both domain and range of full-precision data, promoting the upper bound of quantizer with more quantization parameters. Besides, we utilize knowledge distillation mechanism to adapt the distribution of full-precision data, turning it into more quantization friendly. We demonstrate the effectiveness of our method on CIFAR-10 and ImageNet dataset with various network architectures. Experimental results show that our method bridge the gap between quantized and full-precision models.