Foteini Strati

DéjàVu: KV-cache Streaming for Fast, Fault-tolerant Generative LLM Serving

Distributed LLM serving is costly and often underutilizes hardware accelerators due to three key challenges: bubbles in pipeline-parallel deployments caused by the bimodal latency of prompt and token processing, GPU memory overprovisioning, and long recovery times in case of failures. DéjàVu address…