One-Shot Talking Face Generation with Expression Editing

Abstract

The task of talking face generation has advanced significantly with deep learning, enhancing applications from virtual assistants to animated movies. However, current algorithms face challenges in controlling facial expressions and avoiding unnatural, rigid movements, limiting their realism and effectiveness. Our work introduces a novel approach to improve control over facial expressions and overall fluidity. We use emotion labels to guide audio generation and corresponding facial landmarks, combined with ControlNet for smooth transitions. This method ensures synchronized facial landmarks that reflect desired emotional states and enhances the natural flow of facial movements. Our experiments demonstrate precise emotional control and high-quality visual outputs.