Xinference: Making Large Model Serving Easy
Weizheng Lu, Lingfeng Xiong, Feng Zhang, Xuye Qin, Yueguo Chen
Abstract
The proliferation of open-source large models necessitates dedicated tools for deployment and accessibility. To mitigate the complexities of model serving, we develop Xinference, an open-source library designed to simplify the deployment and management of large models. Xinference effectively simplifies deployment complexities for users by (a) preventing users from writing code and providing built-in support for various models and OpenAI-compatible APIs; (b) enabling full model serving lifecycle management; (c) guaranteeing efficient and scalable inference and achieving high throughput and low latency. In comparative experiments with similar products like BentoML and Ray Serve, Xinference outperforms these tools and offers superior ease of use.Xinference is available at https://github.com/xorbitsai/inference.
BibTeX
@inproceedings{lu-etal-2024-xinference,
title = "Xinference: Making Large Model Serving Easy",
author = "Lu, Weizheng and
Xiong, Lingfeng and
Zhang, Feng and
Qin, Xuye and
Chen, Yueguo",
editor = "Hernandez Farias, Delia Irazu and
Hope, Tom and
Li, Manling",
booktitle = "Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: System Demonstrations",
month = nov,
year = "2024",
address = "Miami, Florida, USA",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2024.emnlp-demo.30/",
doi = "10.18653/v1/2024.emnlp-demo.30",
pages = "291--300"
}