Accurate KV Cache Eviction via Anchor Direction Projection for Efficient LLM Inference
Key-Value (KV) cache eviction---which retains the KV pairs of the most important tokens while discarding less important ones---is a critical technique for optimizing both memory usage and inference latency in large language models (LLMs). However, existing approaches often rely on simple heuristics-…