2025
DON’T STOP ME NOW: EMBEDDING BASED SCHEDULING FOR LLMS
ICLR 2025poster
Efficient scheduling is crucial for interactive Large Language Model (LLM) applications, where low request completion time directly impacts user engagement. Size-based scheduling algorithms like Shortest Remaining Process Time (SRPT) aim to reduce average request completion time by leveraging known…