Sentiment-Aware Automatic Speech Recognition Pre-Training for Enhanced Speech Emotion Recognition

Ayoub Ghriss; Bo Yang; Viktor Rozgic; Elizabeth Shriberg; Chao Wang

ICASSP 2022accepted0 citations

Sentiment-Aware Automatic Speech Recognition Pre-Training for Enhanced Speech Emotion Recognition

Ayoub Ghriss, Bo Yang, Viktor Rozgic, Elizabeth Shriberg, Chao Wang

Abstract

We propose a novel multi-task pre-training method for Speech Emotion Recognition (SER). We pre-train SER model simultaneously on Automatic Speech Recognition (ASR) and sentiment classification tasks to make the acoustic ASR model more "emotion aware". We generate targets for the sentiment classification using text-to-sentiment model trained on publicly available data. Finally, we fine-tune the acoustic ASR on emotion annotated speech data. We evaluated the proposed approach on MSP-Podcast dataset, where we achieved the best reported concordance correlation coefficient (CCC) of 0.41 for valence prediction.

BibTeX

@inproceedings{icassp2022_sentimentawareau,
  title = {Sentiment-Aware Automatic Speech Recognition Pre-Training for Enhanced Speech Emotion Recognition},
  author = {Ayoub Ghriss and Bo Yang and Viktor Rozgic and Elizabeth Shriberg and Chao Wang},
  booktitle = {ICASSP 2022},
  year = {2022}
}