Advancing Streaming ASR with Chunk-wise Attention and Trans-chunk Selective State Spaces

Abstract

This paper explores enhancing streaming speech recognition through the integration of chunk-wise attention and selective state space models (SSMs). The proposed framework replaces the quadratic complexity of attention-based context incorporation with a fully recurrent module based on selective SSMs. Leveraging its constant-time inference and selective reasoning capability, our recurrent module efficiently takes advantage of the entire history across chunks, while conventional chunk-wise encoders are typically constrained to a fixed-length left context to ensure real-time performance. Meanwhile, our within-chunk attention module is designed to focus exclusively on local features within short speech segments, thereby complementing the autoregressive behavior of the SSM. Experimental results from RNN Transducer-based streaming speech recognition indicate that the recurrent and attention modules work synergistically, yielding lower error rates and reduced runtime costs compared to baseline Conformer systems.