LiuWei Liu

Predicting Future KV Utility: Global Combinatorial Optimization for Task-Agnostic KV Cache Eviction

Given the quadratic complexity of attention, KV cache eviction is vital to accelerate model inference. Current KV cache eviction methods typically rely on instantaneous heuristic metrics, implicitly assuming that score magnitudes are consistent proxies for importance across all heads. However, this …