2025
RocketKV: Accelerating Long-Context LLM Inference via Two-Stage KV Cache Compression
ICML 2025poster
Transformer-based Large Language Models rely critically on the KV cache to efficiently handle extended contexts during the decode phase. Yet, the size of the KV cache grows proportionally with the input length, burdening both memory bandwidth and capacity as decoding progresses. To address this chal…