Mixed Precision Neural Quantization with Multi-Objective Bayesian Optimization for on-Device Deployment

Abstract

Mixed-precision quantization has emerged as a solution in recent times for accurate inference of Deep Neural Networks on edge. However the prior-art is far from being deployable on embedded devices due to various practical limitations. In this work, a pipeline is designed that a) performs layer-wise assignment of bit-precisions, b) builds the quantized graph, c) deploys the graph on the embedded device (Galaxy S23), and d) measures the accuracy and on-device performance. This pipeline is optimized using multi-objective Bayesian Optimization for simultaneously maximizing the accuracy and minimizing the on-device inference time, resulting in a Pareto list. The best configuration among them resulted in 3.16, 2.8, and 2.57 times model compression, 31%, 26% and 18% latency improvement and -0.01, 0.27, and 0.08 accuracy drop for ResNet18, MobileNetV2, and InceptionV3, respectively, on ImageNet, establishing a new benchmark in mixed-precision quantization.