ESPnet-SpeechLM: An Open Speech Language Model Toolkit
Jinchuan Tian, Jiatong Shi, William Chen, Siddhant Arora, Yoshiki Masuyama, Takashi Maekaku, Yihan Wu, Junyi Peng
Abstract
We present ESPnet-SpeechLM, an open toolkit designed to democratize the development of speech language models (SpeechLMs) and voice-driven agentic applications. The toolkit standardizes speech processing tasks by framing them as universal sequential modeling problems, encompassing a cohesive workflow of data preprocessing, pre-training, inference, and task evaluation. With ESPnet-SpeechLM, users can easily define task templates and configure key settings, enabling seamless and streamlined SpeechLM development. The toolkit ensures flexibility, efficiency, and scalability by offering highly configurable modules for every stage of the workflow. To illustrate its capabilities, we provide multiple use cases demonstrating how competitive SpeechLMs can be constructed with ESPnet-SpeechLM, including a 1.7B-parameter model pre-trained on both text and speech tasks, across diverse benchmarks. The toolkit and its recipes are fully transparent and reproducible at: https://github.com/espnet/espnet/tree/speechlm.
BibTeX
@inproceedings{tian-etal-2025-espnet,
title = "{ESP}net-{S}peech{LM}: An Open Speech Language Model Toolkit",
author = "Tian, Jinchuan and
Shi, Jiatong and
Chen, William and
Arora, Siddhant and
Masuyama, Yoshiki and
Maekaku, Takashi and
Wu, Yihan and
Peng, Junyi and
Bharadwaj, Shikhar and
Zhao, Yiwen and
Cornell, Samuele and
Peng, Yifan and
Yue, Xiang and
Yang, Chao-Han Huck and
Neubig, Graham and
Watanabe, Shinji",
editor = "Dziri, Nouha and
Ren, Sean (Xiang) and
Diao, Shizhe",
booktitle = "Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (System Demonstrations)",
month = apr,
year = "2025",
address = "Albuquerque, New Mexico",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2025.naacl-demo.12/",
pages = "116--124",
ISBN = "979-8-89176-191-9"
}