Transformer-S2A: Robust and Efficient Speech-to-Animation

Liyang Chen; Zhiyong Wu; Jun Ling; Runnan Li; Xu Tan; Sheng Zhao

ICASSP 2022accepted0 citations

Transformer-S2A: Robust and Efficient Speech-to-Animation

Liyang Chen, Zhiyong Wu, Jun Ling, Runnan Li, Xu Tan, Sheng Zhao

Abstract

We propose a novel robust and efficient Speech-to-Animation (S2A) approach for synchronized facial animation generation in human-computer interaction. Compared with conventional approaches, the proposed approach utilizes phonetic posteriorgrams (PPGs) of spoken phonemes as input to ensure the cross-language and cross-speaker ability, and introduces corresponding prosody features (i.e. pitch and energy) to further enhance the expression of generated animation. Mixture-of-experts (MOE)-based Transformer is employed to better model contextual information while provide significant optimization on computation efficiency. Experiments demonstrate the effectiveness of the proposed approach on both objective and subjective evaluation with 17× inference speedup compared with the state-of-the-art approach.

BibTeX

@inproceedings{icassp2022_transformers2aro,
  title = {Transformer-S2A: Robust and Efficient Speech-to-Animation},
  author = {Liyang Chen and Zhiyong Wu and Jun Ling and Runnan Li and Xu Tan and Sheng Zhao},
  booktitle = {ICASSP 2022},
  year = {2022}
}