2026
THETA: Threshold-Based Exclusive Batching for Memory-Bandwidth-Constrained LLM Inference
ICML 2026poster
Chunked prefill has become the dominant scheduling strategy for large language model (LLM) inference, interleaving prefill and decode operations to improve GPU utilization. However, this approach does not universally outperform exclusive batching: on bandwidth-constrained GPUs, mixed batches can int…