Audio-Synchronized Visual Animation
Lin Zhang, Shentong Mo, Yijing Zhang, Pedro Morgado*
Abstract
"Current visual generation methods can produce high-quality videos guided by text prompts. However, effectively controlling object dynamics remains a challenge. This work explores audio as a cue to generate temporally synchronized image animations. We introduce Audio-Synchronized Visual Animation (), a task that aims to animate a static image of an object with motions temporally guided by audio clips. To this end, we present , a dataset curated from VGGSound with videos featuring synchronized audio-visual events across 15 categories. We also present a diffusion model, , capable of generating audio-guided animations. Extensive evaluations validate as a reliable benchmark for synchronized generation and demonstrate our model’s superior performance. We further explore ’s potential in a variety of audio-synchronized generation tasks, from generating full videos without a base image to controlling object motions with various sounds. We hope our established benchmark can open new avenues for controllable visual generation."
BibTeX
@inproceedings{eccv2024_audiosynchronize,
title = {Audio-Synchronized Visual Animation},
author = {Lin Zhang and Shentong Mo and Yijing Zhang and Pedro Morgado*},
booktitle = {ECCV 2024},
year = {2024}
}