Exploiting Acoustic and Lexical Properties of Phonemes to Recognize Valence from Speech
Biqiao Zhang, Soheil Khorram, Emily Mower Provost
Abstract
Emotions modulate speech acoustics as well as language. The latter influences the sequences of phonemes that are produced, which in turn further modulate the acoustics. Therefore, phonemes impact emotion recognition in two ways: (1) they introduce an additional source of variability in speech signals and (2) they provide information about the emotion expressed in speech content. Previous work in speech emotion recognition has considered (1) or (2), individually. In this paper, we investigate how we can jointly consider both factors to improve the prediction of emotional valence (positive vs. negative), and the relationship between improved prediction and the emotion elicitation process (e.g., fixed script, improvisation, natural interaction). We present a network that exploits both the acoustic and the lexical properties of phonetic information using multi-stage fusion. Our results on the IEMOCAP and MSP-Improv datasets show that our approach outperforms systems that either do not consider the influence of phonetic information or that only consider a single aspect of this influence.
BibTeX
@inproceedings{icassp2019_exploitingacoust,
title = {Exploiting Acoustic and Lexical Properties of Phonemes to Recognize Valence from Speech},
author = {Biqiao Zhang and Soheil Khorram and Emily Mower Provost},
booktitle = {ICASSP 2019},
year = {2019}
}