Baotong Lu

RetrievalAttention: Accelerating Long-Context LLM Inference via Vector Retrieval

Transformer-based Large Language Models (LLMs) have become increasingly important. However, scaling LLMs to longer contexts incurs slow inference speed and high GPU memory consumption for caching key-value (KV) vectors. This paper presents RetrievalAttention, a training-free approach to both acceler…