SLBERT: A Novel Pre-Training Framework for Joint Speech and Language Modeling

Abstract

We propose SLBERT (Speech and Language pre-training framework for BERT), an end-to-end trainable framework for learning joint representations of speech and language modalities. We enhance the well-known BERT architecture to provide a dual-stream multimodal architecture that processes both speech and language input. To enable effective information exchange between the two modalities, we introduce a novel attention fusion mechanism via AF-Blocks. To acquire robust contrastive representations for speech and language processing applications, we pre-train SLBERT on three auxiliary tasks: Masked Language Modeling, Masked Speech Modeling, and Speech-Language Matching. We evaluate our proposed model on two well-known multimodal tasks: intent classification and sentiment analysis. Our model achieves state-of-the-art results on both benchmarks while surpassing even larger baselines.