SwapTalk: Audio-Driven Talking Face Generation with One-Shot Customization in Latent Space
Zeren Zhang, Haibo Qin, Jiayu Huang, Jo-Ku Cheng, Yixin Li, Hui Lin, Yitao Duan, Jinwen Ma
Abstract
Combining face-swapping with lip synchronization offers a cost-effective solution for generating customized talking faces. However, directly cascading existing models can introduce significant interference and reduce video clarity due to limited interaction space in the low-level RGB domain. To solve this, we propose SwapTalk, a unified framework that performs face-swapping and lip synchronization within the same latent VQ-embedding space, known for its editability and fidelity. We enhance generalization to unseen identities with identity loss in the face-swapping module and improve synchronization quality with expert discriminator supervision. To better approximate real-world applications, we expand the evaluation scope to asynchronous audio-video scenarios. Furthermore, we introduce a novel identity consistency metric to more comprehensively assess the identity consistency over time series in generated facial videos. Experiments on HDTF show that SwapTalk outperforms existing methods in video quality, lip synchronization accuracy, face-swapping fidelity, and identity consistency.
BibTeX
@inproceedings{icassp2025_swaptalkaudiodri,
title = {SwapTalk: Audio-Driven Talking Face Generation with One-Shot Customization in Latent Space},
author = {Zeren Zhang and Haibo Qin and Jiayu Huang and Jo-Ku Cheng and Yixin Li and Hui Lin and Yitao Duan and Jinwen Ma},
booktitle = {ICASSP 2025},
year = {2025}
}