2026
FAFO: Lossy KV Cache Compression for Lossless Inference Acceleration via Draftless Fumble Decoding
ICML 2026poster
Lossy KV cache compression is a well-explored subfield of machine learning efficiency, with improved latency being one of its major gains. However, lossy compression techniques can fumble from time to time, exhibiting various — and often catastrophic — failure patterns that are not only difficult to…