Speech-N-LlaMA: Improving Speech LLMs with Multi-Pass Training
Amit Kumar Singh Yadav, Gil Keren, Desh Raj, Wei Zhou, Junteng Jia, Ke Li, Ying Xu, Chunyang Wu
Abstract
Speech LLMs use speech embeddings as the prompt to a Large Language Model (LLM) and generate human readable text for the speech signal in an autoregressive manner. Teacher-forcing is a common approach used for training Speech LLMs, which is dissimilar to the procedure used during inference, creating a gap between training and inference known as exposure bias. To mitigate exposure bias, we propose Speech-N-LlaMA. Contrary to the existing Speech LLMs that have single pass through the LLM during training, Speech-N-LlaMA incorporates multi-pass training. Through multiple passes, Speech-N-LlaMA mitigates exposure bias and uses the error correction capability of LLM to improve the performance of Speech LLMs. We propose an N-pass loss and utterance level temperature sampling in Speech-N-LlaMA to achieve this. We evaluate four different model sizes on three benchmarks, and show up to 18% relative improvement in Word Error Rate (WER) compared to baseline Speech LLM, while not incurring any additional compute during inference.
BibTeX
@inproceedings{icassp2025_speechnllamaimpr,
title = {Speech-N-LlaMA: Improving Speech LLMs with Multi-Pass Training},
author = {Amit Kumar Singh Yadav and Gil Keren and Desh Raj and Wei Zhou and Junteng Jia and Ke Li and Ying Xu and Chunyang Wu and Jay Mahadeokar and Ozlem Kalinli},
booktitle = {ICASSP 2025},
year = {2025}
}