Learning Individual Styles of Conversational Gesture

Shiry Ginosar; Amir Bar; Gefen Kohavi; Caroline Chan; Andrew Owens; Jitendra Malik

CVPR 2019poster403 citations

Learning Individual Styles of Conversational Gesture

Shiry Ginosar, Amir Bar, Gefen Kohavi, Caroline Chan, Andrew Owens, Jitendra Malik

Abstract

Human speech is often accompanied by hand and arm gestures. We present a method for cross-modal translation from "in-the-wild" monologue speech of a single speaker to their conversational gesture motion. We train on unlabeled videos for which we only have noisy pseudo ground truth from an automatic pose detection system. Our proposed model significantly outperforms baseline methods in a quantitative comparison. To support research toward obtaining a computational understanding of the relationship between gesture and speech, we release a large video dataset of person-specific gestures.

BibTeX

@inproceedings{cvpr2019_learningindividu,
  title = {Learning Individual Styles of Conversational Gesture},
  author = {Shiry Ginosar and Amir Bar and Gefen Kohavi and Caroline Chan and Andrew Owens and Jitendra Malik},
  booktitle = {CVPR 2019},
  year = {2019}
}