CoVAR: Co-Generation of Video and Action for Robotic Manipulation Via Multi-Modal Diffusion

Abstract

We present a method to generate video–action pairs that follow text instructions, starting from an initial image observation and the robot’s joint states. Our approach automatically provides action labels for video diffusion mod- els, overcoming the common lack of action annotations and enabling their full use for robotic policy learning. Existing methods either adopt two-stage pipelines, which limit tightly coupled cross-modal information sharing, or rely on adapting a single-modal diffusion model for joint distribution that cannot fully leverage pretrained video knowledge. To overcome these limitations, we (1) extend a pretrained video diffusion model with a parallel, dedicated action diffusion model that preserves pretrained knowledge, (2) introduce a Bridge Attention mecha- nism to enable effective cross-modal interaction, and (3) design an action refinement module to convert coarse actions into precise controls for low-resolution datasets. Experiments on public benchmarks and real-world datasets demonstrate that our method produces higher-quality videos, more accurate ac- tions, and significantly outperforms existing baselines, offering a scalable framework for leveraging large-scale video data for robotic learning