2026
DuetServe: Harmonizing Prefill and Decode for LLM Serving via Adaptive GPU Multiplexing
ICML 2026poster
Modern LLM serving systems must sustain high throughput while meeting strict latency SLOs across two distinct inference phases: compute-intensive prefill and memory-bound decode phases. Existing approaches either (1) aggregate both phases on shared GPUs, leading to interference between prefill and d…