Yaosheng Fu

RocketKV: Accelerating Long-Context LLM Inference via Two-Stage KV Cache Compression

Transformer-based Large Language Models rely critically on the KV cache to efficiently handle extended contexts during the decode phase. Yet, the size of the KV cache grows proportionally with the input length, burdening both memory bandwidth and capacity as decoding progresses. To address this chal…