CSR:Achieving 1 Bit Key-Value Cache via Sparse Representation
Hongxuan Zhang, Yao Zhao, Jiaqi Zheng, Chenyi Zhuang, Jinjie Gu, Guihai Chen
Abstract
The emergence of long-context text applications utilizing large language models (LLMs) has presented significant scalability challenges, particularly in memory footprint. The linear growth of the Key-Value (KV) cache, which stores attention keys and values to reduce redundant computations, can significantly increase memory usage and may prevent models from functioning properly in memory-constrained environments. To address this issue, we propose a novel approach called Cache Sparse Representation (CSR), which converts the KV cache by transforming the dense Key-Value cache tensor into sparse indexes and weights, offering a more memory-efficient representation during LLM inference. Furthermore, we introduce NeuralDict, a novel neural network-based method to automatically generate the dictionary used in our sparse representation. Our extensive experiments demonstrate that CSR matches the performance of state-of-the-art KV cache quantization algorithms while ensuring robust functionality in memory-constrained environments.
BibTeX
@article{Zhang_Zhao_Zheng_Zhuang_Gu_Chen_2025, title={CSR:Achieving 1 Bit Key-Value Cache via Sparse Representation}, volume={39}, url={https://ojs.aaai.org/index.php/AAAI/article/view/34779}, DOI={10.1609/aaai.v39i24.34779}, abstractNote={The emergence of long-context text applications utilizing large language models (LLMs) has presented significant scalability challenges, particularly in memory footprint. The linear growth of the Key-Value (KV) cache, which stores attention keys and values to reduce redundant computations, can significantly increase memory usage and may prevent models from functioning properly in memory-constrained environments. To address this issue, we propose a novel approach called Cache Sparse Representation (CSR), which converts the KV cache by transforming the dense Key-Value cache tensor into sparse indexes and weights, offering a more memory-efficient representation during LLM inference. Furthermore, we introduce NeuralDict, a novel neural network-based method to automatically generate the dictionary used in our sparse representation. Our extensive experiments demonstrate that CSR matches the performance of state-of-the-art KV cache quantization algorithms while ensuring robust functionality in memory-constrained environments.}, number={24}, journal={Proceedings of the AAAI Conference on Artificial Intelligence}, author={Zhang, Hongxuan and Zhao, Yao and Zheng, Jiaqi and Zhuang, Chenyi and Gu, Jinjie and Chen, Guihai}, year={2025}, month={Apr.}, pages={25860-25867} }