2025
DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Heads
ICLR 2025poster
Deploying long-context large language models (LLMs) is essential but poses significant computational and memory challenges. Caching all Key and Value (KV) states across all attention heads consumes substantial memory. Existing KV cache pruning methods either damage the long-context capabilities of L…