SIDGAN: High-Resolution Dubbed Video Generation via Shift-Invariant Learning
Urwa Muaz, Wondong Jang, Rohun Tripathi, Santhosh Mani, Wenbin Ouyang, Ravi Teja Gadde, Baris Gecer, Sergio Elizondo
Abstract
Dubbed video generation aims to accurately synchronize mouth movements of a given facial video with driving audio while preserving identity and scene-specific visual dynamics, such as head pose and lighting. Despite the accurate lip generation of previous approaches that adopts a pretrained audio-video synchronization metric as an objective function, called Sync-Loss, extending it to high-resolution videos was challenging due to shift biases in the loss landscape that inhibit tandem optimization of Sync-Loss and visual quality, leading to a loss of detail. To address this issue, we introduce shift-invariant learning, which generates photo-realistic high-resolution videos with accurate Lip-Sync. Further, we employ a pyramid network with coarse-to-fine image generation to improve stability and lip syncronization. Our model outperforms state-of-the-art methods on multiple benchmark datasets, including AVSpeech, HDTF, and LRW, in terms of photo-realism, identity preservation, and Lip-Sync accuracy.
BibTeX
@inproceedings{iccv2023_sidganhighresolu,
title = {SIDGAN: High-Resolution Dubbed Video Generation via Shift-Invariant Learning},
author = {Urwa Muaz and Wondong Jang and Rohun Tripathi and Santhosh Mani and Wenbin Ouyang and Ravi Teja Gadde and Baris Gecer and Sergio Elizondo and Reza Madad and Naveen Nair},
booktitle = {ICCV 2023},
year = {2023}
}