Wenhai Lin

AugServe: Adaptive Request Scheduling for Augmented Large Language Model Inference Serving

Augmented large language models (LLMs) that invoke external calls are increasingly prevalent in inference serving. However, such augmentations pose significant challenges to inference efficiency under strict Service-Level Objectives (SLOs). Existing inference systems are agnostic to the dynamic exec…