BEYOND VIDEO-TO-SFX: VIDEO TO AUDIO SYNTHESIS WITH ENVIRONMENTALLY AWARE SPEECH
Xinlei Niu, Jianbo Ma, Xiangyu Zhang, Charles Partick Martin
Abstract
The generation of realistic, context-aware audio is important in real-world applications such as video game development. While existing video-to-audio (V2A) methods mainly focus on Foley sound generation, they struggle to produce intelligible speech. Meanwhile, current environmental speech synthesis approaches remain text-driven and fail to temporally align with dynamic video content. In this paper, we propose Beyond Video-to-SFX (BVS), a method to generate synchronized audio with environmentally aware intelligible speech for given videos. We introduce a two-stage modeling method: (1) stage one is a video-guided audio semantic (V2AS) model to predict unified audio semantic tokens conditioned on phonetic cues; (2) stage two is a video-conditioned semantic-to-acoustic (VS2A) model that refines semantic tokens into detailed acoustic tokens. Experiments demonstrate the effectiveness of BVS in scenarios such as video-to-context-aware speech synthesis and immersive audio background conversion, with ablation studies further validating our design. Our demonstration is available at~\href{https://xinleiniu.github.io/BVS-demo/}{BVS-Demo}.
BibTeX
@inproceedings{icassp2026_beyondvideotosfx,
title = {BEYOND VIDEO-TO-SFX: VIDEO TO AUDIO SYNTHESIS WITH ENVIRONMENTALLY AWARE SPEECH},
author = {Xinlei Niu and Jianbo Ma and Xiangyu Zhang and Charles Partick Martin},
booktitle = {ICASSP 2026},
year = {2026}
}