2026
Randomization Boosts KV Caching, Learning Balances Query Load: A Joint Perspective
ICLR 2026poster
KV caching is a fundamental technique for accelerating Large Language Model (LLM) inference by reusing key-value (KV) pairs from previous queries, but its effectiveness under limited memory is highly sensitive to the eviction policy. The default Least Recently Used (LRU) eviction algorithm struggle…