MetaMix: Meta-State Precision Searcher for Mixed-Precision Activation Quantization
Han-Byul Kim, Joo Hyung Lee, Sungjoo Yoo, Hong-Seok Kim
Abstract
Mixed-precision quantization of efficient networks often suffer from activation instability encountered in the exploration of bit selections. To address this problem, we propose a novel method called MetaMix which consists of bit selection and weight training phases. The bit selection phase iterates two steps, (1) the mixed-precision-aware weight update, and (2) the bit-search training with the fixed mixed-precision-aware weights, both of which combined reduce activation instability in mixed-precision quantization and contribute to fast and high-quality bit selection. The weight training phase exploits the weights and step sizes trained in the bit selection phase and fine-tunes them thereby offering fast training. Our experiments with efficient and hard-to-quantize networks, i.e., MobileNet v2 and v3, and ResNet-18 on ImageNet show that our proposed method pushes the boundary of mixed-precision quantization, in terms of accuracy vs. operations, by outperforming both mixed- and single-precision SOTA methods.
BibTeX
@article{Kim_Lee_Yoo_Kim_2024, title={MetaMix: Meta-State Precision Searcher for Mixed-Precision Activation Quantization}, volume={38}, url={https://ojs.aaai.org/index.php/AAAI/article/view/29212}, DOI={10.1609/aaai.v38i12.29212}, abstractNote={Mixed-precision quantization of efficient networks often suffer from activation instability encountered in the exploration of bit selections. To address this problem, we propose a novel method called MetaMix which consists of bit selection and weight training phases. The bit selection phase iterates two steps, (1) the mixed-precision-aware weight update, and (2) the bit-search training with the fixed mixed-precision-aware weights, both of which combined reduce activation instability in mixed-precision quantization and contribute to fast and high-quality bit selection. The weight training phase exploits the weights and step sizes trained in the bit selection phase and fine-tunes them thereby offering fast training. Our experiments with efficient and hard-to-quantize networks, i.e., MobileNet v2 and v3, and ResNet-18 on ImageNet show that our proposed method pushes the boundary of mixed-precision quantization, in terms of accuracy vs. operations, by outperforming both mixed- and single-precision SOTA methods.}, number={12}, journal={Proceedings of the AAAI Conference on Artificial Intelligence}, author={Kim, Han-Byul and Lee, Joo Hyung and Yoo, Sungjoo and Kim, Hong-Seok}, year={2024}, month={Mar.}, pages={13132-13141} }