Audio Diffusion with Large Language Models

Yinghui Huang; Kyle Kastner; Kartik Audhkhasi; Bhuvana Ramabhadran; Andrew Rosenberg

ICASSP 2025accepted0 citations

Audio Diffusion with Large Language Models

Yinghui Huang, Kyle Kastner, Kartik Audhkhasi, Bhuvana Ramabhadran, Andrew Rosenberg

Abstract

In this paper, we explore an alternate approach to the popular method of using large language models (LLMs) as a second decoder for Automated Speech Recognition (ASR) and speech understanding tasks. We propose to employ diffusion networks to generate a correction signal that can be applied on the original input audio features to improve performance. Specifically, the diffusion network is trained to predict the gradient of any ASR objective with respect to the input audio features conditioned on LLM embeddings. Our experiments are conducted on public corpora, namely, Librispeech and Common Voice. We show that the diffusion model is able to improve ASR performance on noisy and accented speech, with the addition of knowledge from the LLM, and also helps improve generalization to out-of-domain test sets.

BibTeX

@inproceedings{icassp2025_audiodiffusionwi,
  title = {Audio Diffusion with Large Language Models},
  author = {Yinghui Huang and Kyle Kastner and Kartik Audhkhasi and Bhuvana Ramabhadran and Andrew Rosenberg},
  booktitle = {ICASSP 2025},
  year = {2025}
}