XLSR-Transducer: Streaming ASR for Self-Supervised Pretrained Models
Shashi Kumar, Srikanth R. Madikeri, Juan Zuluaga-Gomez, Esaú Villatoro-Tello, Iuliia Thorbecke, Petr Motlícek, Manjunath K. E, Aravind Ganapathiraju
Abstract
Self-supervised pretrained models exhibit competitive performance in automatic speech recognition (ASR) on finetuning, even with limited in-domain supervised data. However, popular pretrained models are not suitable for streaming ASR because they are trained with full attention context. In this paper, we introduce XLSR-Transducer, where the XLSR-53 model is used as encoder in transducer setup. Our experiments on the AMI dataset reveal that the XLSR-Transducer achieves 4% absolute WER improvement over Whisper large-v2 and 8% over a Zipformer transducer model trained from scratch. To enable streaming capabilities, we investigate different attention masking patterns in the self-attention computation of transformer layers within the XLSR-53 model. We validate XLSR-Transducer on AMI and 5 languages from CommonVoice under low-resource scenarios. Finally, with the introduction of attention sinks, we reduce the left context by half while achieving a relative 12% improvement in WER.
BibTeX
@inproceedings{icassp2025_xlsrtransducerst,
title = {XLSR-Transducer: Streaming ASR for Self-Supervised Pretrained Models},
author = {Shashi Kumar and Srikanth R. Madikeri and Juan Zuluaga-Gomez and Esaú Villatoro-Tello and Iuliia Thorbecke and Petr Motlícek and Manjunath K. E and Aravind Ganapathiraju},
booktitle = {ICASSP 2025},
year = {2025}
}