SmartNet: One-shot Talking Head Synthesis via Subtle Motion and Appearance Compensation

Abstract

One-shot talking head synthesis aims to animate a source person’s portrait with driving video sequences. Recent facial keypoint-based methods have achieved remarkable animation performance and produced high-quality results. However, it remains challenging to perform cross-identity face reenactment by transferring subtle facial motions with correct geometry and appearance. To break the above limitations, in this paper, we propose a subtle motion compensation network to recover correct facial expressions by leveraging the decoupled 3D Morphable Model (3DMM) coefficient. In addition, to generate faithful animation results, a facial appearance feature memory bank is designed to learn accurate facial features and better recover the appearance. Experimental results have demonstrated that our proposed model can outperform state-of-the-art methods by generating faithful videos with correct subtle motion transfer and consistent identity preserving.