Expressive Talking Head Generation With Granular Audio-Visual Control
Borong Liang, Yan Pan, Zhizhi Guo, Hang Zhou, Zhibin Hong, Xiaoguang Han, Junyu Han, Jingtuo Liu
Abstract
Generating expressive talking heads is essential for creating virtual humans. However, existing one- or few-shot methods focus on lip-sync and head motion, ignoring the emotional expressions that make talking faces realistic. In this paper, we propose the Granularly Controlled Audio-Visual Talking Heads (GC-AVT), which controls lip movements, head poses, and facial expressions of a talking head in a granular manner. Our insight is to decouple the audio-visual driving sources through prior-based pre-processing designs. Detailedly, we disassemble the driving image into three complementary parts including: 1) a cropped mouth that facilitates lip-sync; 2) a masked head that implicitly learns pose; and 3) the upper face which works corporately and complementarily with a time-shifted mouth to contribute the expression. Interestingly, the encoded features from the three sources are integrally balanced through reconstruction training. Extensive experiments show that our method generates expressive faces with not only synced mouth shapes, controllable poses, but precisely animated emotional expressions as well.
BibTeX
@inproceedings{cvpr2022_expressivetalkin,
title = {Expressive Talking Head Generation With Granular Audio-Visual Control},
author = {Borong Liang and Yan Pan and Zhizhi Guo and Hang Zhou and Zhibin Hong and Xiaoguang Han and Junyu Han and Jingtuo Liu and Errui Ding and Jingdong Wang},
booktitle = {CVPR 2022},
year = {2022}
}