Seong Hoon Seo

DP-LLM: Runtime Model Adaptation with Dynamic Layer-wise Precision Assignment

How can we effectively handle queries for on-device large language models (LLMs) with varying runtime constraints, such as latency and accuracy? Multi-scale quantization addresses this challenge by enabling memory-efficient runtime model adaptation of LLMs through the overlaying of multiple model va…