RaBitQCache: Rotated Binary Quantization for KVCache in Long Context LLM Inference
Wenhao Li, Jinhao Dong, Hailin Zhang, Wenhang Shi, WEI LU, Xiaoyong Du
Abstract
Long-context Large Language Model inference is severely bottlenecked by the massive Key-Value (KV) cache, yet existing sparse attention methods often suffer from static fixed-budget (Top-k) retrieval or rely on proxy scores that are computationally expensive and biased. To address these limitations, we propose RaBitQCache, a novel sparse attention framework that utilizes randomized rotated binary quantization and high-throughput binary-INT4 arithmetic to efficiently estimate attention weights. Our proxy score serves as an unbiased estimator with a proven error bound, enabling adaptive Top-p retrieval that dynamically adjusts the token budget based on actual attention sparsity. We further implement a hardware-aware system with asynchronous pipelining and lazy updates to mask overhead. Evaluations demonstrate that RaBitQCache significantly accelerates inference and reduces memory I/O while preserving generation quality compared to state-of-the-art baselines.
BibTeX
@inproceedings{
li2026rabitqcache,
title={RaBit{QC}ache: Rotated Binary Quantization for {KVC}ache in Long Context {LLM} Inference},
author={Wenhao Li and Jinhao Dong and Hailin Zhang and Wenhang Shi and WEI LU and Xiaoyong Du},
booktitle={Forty-third International Conference on Machine Learning},
year={2026},
url={https://openreview.net/forum?id=zc8ZDMJmKB}
}