Liuyue

ChunkKV: Semantic-Preserving KV Cache Compression for Efficient Long-Context LLM Inference

Large Language Models (LLMs) require significant GPU memory when processing long texts, with the key value (KV) cache consuming up to 70\% of total memory during inference. Although existing compression methods reduce memory by evaluating the importance of individual tokens, they overlook critical s…