2024
CHAI: Clustered Head Attention for Efficient LLM Inference
ICML 2024poster
Large Language Models (LLMs) with hundreds of billions of parameters have transformed the field of machine learning. However, serving these models at inference time is both compute and memory intensive, where a single request can require multiple GPUs and tens of Gigabytes of memory. Multi-head atte…