2024
Compressed Context Memory for Online Language Model Interaction
ICLR 2024poster
This paper presents a context key/value compression method for Transformer language models in online scenarios, where the context continually expands. As the context lengthens, the attention process demands increasing memory and computations, which in turn reduces the throughput of the language mode…