Weifan Jiang

DON’T STOP ME NOW: EMBEDDING BASED SCHEDULING FOR LLMS

Efficient scheduling is crucial for interactive Large Language Model (LLM) applications, where low request completion time directly impacts user engagement. Size-based scheduling algorithms like Shortest Remaining Process Time (SRPT) aim to reduce average request completion time by leveraging known…