QoKV: Comprehending and Surpassing the Hurdles of KV Cache Quantization

Abstract

Large language models (LLMs) have demonstrated outstanding performance in various tasks. However, the memory footprint of the key-value (KV) cache generated during model inference poses significant challenges for efficient model deployment. This paper presents a detailed analysis of the KV cache and identifies potential sources of quantization errors. We find that the attention scores exhibit a strong power-law distribution while showing an additional attention to the initial and recent tokens, and that only a small subset of outlier channels in KV cache span a wide numerical range. Based on these findings, we propose QoKV, an efficient post-training quantization (PTQ) method designed for the KV cache, which facilitates accurate low-bit quantization through a novel group evaluation strategy and numerical scaling. Extensive experimental results across various tasks demonstrate that our proposed method outperforms existing methods, achieving near-floating-point performance while saving 73.75% of the cache footprint.