Resource-Efficient and Noise-Robust Modality Fusion for Audio-Visual Speech Recognition

Abstract

Resource-efficient audio-visual fusion techniques often struggle to maintain robust performance across varying acoustic noise conditions in speech recognition tasks. This paper introduces a dynamic routing approach for noise-robust audio-visual fusion, which adaptively directs features to noise-specific subnetworks. Unlike attention-based fusion methods that incur quadratic computational complexity, our approach maintains constant complexity, making it suitable for resource-constrained applications. We evaluate our method on the "Lip Reading in the Wild" dataset for isolated (within context) word recognition, augmented with diverse acoustic noise conditions. Results demonstrate significant improvements over feature concatenation and gating baselines of similar computational complexity. In extreme noise (-20 dB), our approach achieves 89.51% mean word accuracy, outperforming the next-best method (bimodal gating) by 0.63% absolute. Notably, it also excels in low-noise scenarios (20 dB), with only a marginal 0.03% absolute accuracy decrease compared to the audio-only system, while the second-best method shows a 0.54% absolute decline. These results demonstrate our method’s effectiveness in mitigating catastrophic fusion and maintaining high performance across diverse acoustic environments, while remaining computationally efficient.