Towards Dialogue Modeling Beyond Text

Tongzi Wu; Yuhao Zhou; Wang Ling; Hojin Yang; Joana Veloso; Lin Sun; Ruixin Huang; Norberto Guimaraes

ICASSP 2023accepted0 citations

Towards Dialogue Modeling Beyond Text

Tongzi Wu, Yuhao Zhou, Wang Ling, Hojin Yang, Joana Veloso, Lin Sun, Ruixin Huang, Norberto Guimaraes

Abstract

In this paper, we model aspects of communication beyond the words that are said. Specifically, we aim to detect interruptions and active listening events, which are important elements in any dialogue. We build a dataset with fine-grained annotations for each category and train multimodal models that take into account all channels in a digital conversation, that is, the video, the audio, and the text. Our experiments show that multimodality is a necessary component in modeling the complexity of the non-textual components of the conversation as different artifacts require different modalities to capture effectively.

BibTeX

@inproceedings{icassp2023_towardsdialoguem,
  title = {Towards Dialogue Modeling Beyond Text},
  author = {Tongzi Wu and Yuhao Zhou and Wang Ling and Hojin Yang and Joana Veloso and Lin Sun and Ruixin Huang and Norberto Guimaraes and Scott Sanner},
  booktitle = {ICASSP 2023},
  year = {2023}
}