Speech Emotion Recognition with Dual-Sequence LSTM Architecture

Jianyou Wang; Michael Xue; Ryan Culhane; Enmao Diao; Jie Ding; Vahid Tarokh

ICASSP 2020accepted0 citations

Speech Emotion Recognition with Dual-Sequence LSTM Architecture

Jianyou Wang, Michael Xue, Ryan Culhane, Enmao Diao, Jie Ding, Vahid Tarokh

Abstract

Speech Emotion Recognition (SER) has emerged as a critical component of the next generation of human-machine interfacing technologies. In this work, we propose a new dual-level model that predicts emotions based on both MFCC features and mel-spectrograms produced from raw audio signals. Each utterance is preprocessed into MFCC features and two mel-spectrograms at different time-frequency resolutions. A standard LSTM processes the MFCC features, while a novel LSTM architecture, denoted as Dual-Sequence LSTM (DS-LSTM), processes the two mel-spectrograms simultaneously. The outputs are later averaged to produce a final classification of the utterance. Our proposed model achieves, on average, a weighted accuracy of 72.7% and an unweighted accuracy of 73.3% - a 6% improvement over current state-of-the-art unimodal models - and is comparable with multimodal models that leverage textual information as well as audio signals.

BibTeX

@inproceedings{icassp2020_speechemotionrec,
  title = {Speech Emotion Recognition with Dual-Sequence LSTM Architecture},
  author = {Jianyou Wang and Michael Xue and Ryan Culhane and Enmao Diao and Jie Ding and Vahid Tarokh},
  booktitle = {ICASSP 2020},
  year = {2020}
}