EMNLP 20250 citations
Cacheback: Speculative Decoding With Nothing But Cache
Abstract
We present Cacheback Decoding, a training-free and model-agnostic speculative decoding method that exploits the locality in language to accelerate Large Language Model (LLM) inference.Cacheback leverages only Least Recently Used (LRU) cache tables of token n-grams to generate draft sequences.Cacheback achieves state-of-the-art performance among comparable methods despite its minimalist design, and its simplicity allows easy integration into existing systems.Cacheback also shows potential for fast adaptation to new domains.
BibTeX
@inproceedings{emnlp2025_cachebackspecula,
title = {Cacheback: Speculative Decoding With Nothing But Cache},
author = {Zhiyao Ma and In Gim and Lin Zhong},
booktitle = {EMNLP 2025},
year = {2025}
}