Poulami Das

Dialogue Without Limits: Constant-Sized KV Caches for Extended Response in LLMs

Autoregressive Transformers rely on Key-Value (KV) caching to accelerate inference. However, the linear growth of the KV cache with context length leads to excessive memory consumption and bandwidth constraints. Existing methods drop distant tokens or compress states in a lossy manner, sacrificing a…