2024
Dynamic Memory Compression: Retrofitting LLMs for Accelerated Inference
ICML 2024poster
Transformers have emerged as the backbone of large language models (LLMs). However, generation remains inefficient due to the need to store in memory a cache of key–value representations for past tokens, whose size scales linearly with the input sequence length and batch size. As a solution, we prop…