2026
Predicting Future KV Utility: Global Combinatorial Optimization for Task-Agnostic KV Cache Eviction
ICML 2026poster
Given the quadratic complexity of attention, KV cache eviction is vital to accelerate model inference. Current KV cache eviction methods typically rely on instantaneous heuristic metrics, implicitly assuming that score magnitudes are consistent proxies for importance across all heads. However, this …