Shining Wu

THETA: Threshold-Based Exclusive Batching for Memory-Bandwidth-Constrained LLM Inference

Chunked prefill has become the dominant scheduling strategy for large language model (LLM) inference, interleaving prefill and decode operations to improve GPU utilization. However, this approach does not universally outperform exclusive batching: on bandwidth-constrained GPUs, mixed batches can int…