2024
Reducing Transformer Key-Value Cache Size with Cross-Layer Attention
NeurIPS 2024poster
Key-value (KV) caching plays an essential role in accelerating decoding for transformer-based autoregressive large language models (LLMs). However, the amount of memory required to store the KV cache can become prohibitive at long sequence lengths and large batch sizes. Since the invention of the tr…