2024
KV Cache is 1 Bit Per Channel: Efficient Large Language Model Inference with Coupled Quantization
NeurIPS 2024poster
Efficient deployment of Large Language Models (LLMs) requires batching multiple requests together to improve throughput. As batch size, context length, or model size increases, the size of key and value (KV) cache quickly becomes the main contributor to GPU memory usage and the bottleneck of inferen…