Demystifying Small Language Models for Edge Deployment
Zhenyan Lu, Xiang Li, Dongqi Cai, Rongjie Yi, Fangming Liu, Wei Liu, Jian Luan, Xiwen Zhang
Abstract
Small language models (SLMs) have emerged as a promising solution for deploying resource-constrained devices, such as smartphones and Web of Things. This work presents the first comprehensive study of over 60 SLMs such as Microsoft Phi and Google Gemma that are publicly accessible. Our findings show that state-of-the-art SLMs outperform 7B models in general tasks, proving their practical viability. However, SLMs’ in-context learning capabilities remain limited, and their efficiency has significant optimization potential. We identify key SLM optimization opportunities, including dynamic task-specific routing, model-hardware co-design, and vocabulary/KV cache compression. Overall, we expect the work to reveal an all-sided landscape of SLMs, benefiting the research community across algorithm, model, system, and hardware levels.
BibTeX
@inproceedings{lu-etal-2025-demystifying,
title = "Demystifying Small Language Models for Edge Deployment",
author = "Lu, Zhenyan and
Li, Xiang and
Cai, Dongqi and
Yi, Rongjie and
Liu, Fangming and
Liu, Wei and
Luan, Jian and
Zhang, Xiwen and
Lane, Nicholas D. and
Xu, Mengwei",
editor = "Che, Wanxiang and
Nabende, Joyce and
Shutova, Ekaterina and
Pilehvar, Mohammad Taher",
booktitle = "Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
month = jul,
year = "2025",
address = "Vienna, Austria",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2025.acl-long.718/",
doi = "10.18653/v1/2025.acl-long.718",
pages = "14747--14764",
ISBN = "979-8-89176-251-0"
}