Zhuoyifan

Distributed LLM Serving on Consumer-Grade GPUs by Reconciling Computation and Communication

Large language models are reshaping internet services. Serving these models is often costly, as it requires multiple high-end GPUs. Consumer-grade GPUs offer cheaper computational power, providing an opportunity for more cost-efficient LLM serving.Prior efforts have explored distributed serving at s