Language-Models-as-a-Service: Overview of a New Paradigm and its Challenges
Emanuele La Malfa, Aleksandar Petrov, Simon Frieder, Christoph Weinhuber, Ryan Burnell, Raza Nazar, Anthony Cohn, Nigel Shadbolt
Abstract
Some of the most powerful language models currently are proprietary systems, accessible only via (typically restrictive) web or software programming interfaces. This is the LanguageModels-as-a-Service (LMaaS) paradigm. In contrast with scenarios where full model access is available, as in the case of open-source models, such closed-off language models present specific challenges for evaluating, benchmarking, and testing them. This paper has two goals: on the one hand, we delineate how the aforementioned challenges act as impediments to the accessibility, reproducibility, reliability, and trustworthiness of LMaaS. We systematically examine the issues that arise from a lack of information about language models for each of these four aspects. We conduct a detailed analysis of existing solutions, put forth a number of recommendations, and highlight directions for future advancements. On the other hand, it serves as a synthesized overview of the licences and capabilities of the most popular LMaaS.
BibTeX
@article{La Malfa_Petrov_Frieder_Weinhuber_Burnell_Nazar_Cohn_Shadbolt_Wooldridge_2025, title={Language-Models-as-a-Service: Overview of a New Paradigm and its Challenges}, volume={39}, url={https://ojs.aaai.org/index.php/AAAI/article/view/35133}, DOI={10.1609/aaai.v39i27.35133}, abstractNote={Some of the most powerful language models currently are proprietary systems, accessible only via (typically restrictive) web or software programming interfaces. This is the LanguageModels-as-a-Service (LMaaS) paradigm. In contrast with scenarios where full model access is available, as in the case of open-source models, such closed-off language models present specific challenges for evaluating, benchmarking, and testing them. This paper has two goals: on the one hand, we delineate how the aforementioned challenges act as impediments to the accessibility, reproducibility, reliability, and trustworthiness of LMaaS. We systematically examine the issues that arise from a lack of information about language models for each of these four aspects. We conduct a detailed analysis of existing solutions, put forth a number of recommendations, and highlight directions for future advancements. On the other hand, it serves as a synthesized overview of the licences and capabilities of the most popular LMaaS.}, number={27}, journal={Proceedings of the AAAI Conference on Artificial Intelligence}, author={La Malfa, Emanuele and Petrov, Aleksandar and Frieder, Simon and Weinhuber, Christoph and Burnell, Ryan and Nazar, Raza and Cohn, Anthony and Shadbolt, Nigel and Wooldridge, Michael}, year={2025}, month={Apr.}, pages={28742-28742} }