Zhisheng Ye

PACE: Proactive Agent-Level Admission Control for Efficient Agentic Batch Inference

Batch inference for agentic workloads stresses the GPU key–value (KV) cache in a sustained and cumulative manner, often causing severe throughput degradation well before memory capacity is exhausted. We identify this phenomenon as middle-phase thrashing, a previously under-characterized pathology in…