Twin-DP3: View-Invariant 3D Diffusion Policy With Digital Twin

Abstract

Learning visuomotor policies with imitation learning from 3D observations is a primary research direction in robotic manipulation, as 3D data inherently captures spatial features critical for action. While many existing methods rely on multiview point cloud fusion, recent studies like 3D Diffusion Policy have shown that using a single-view observation setting with a lightweight point cloud encoder can achieve robust 3D policy learning. This design yields a simpler and more efficient pipeline. However, we observe that such a setting suffers from performance degradation under significant viewpoint changes. To overcome this limitation without requiring extensive multi-view data acquisition, we propose a digital twin-based real-time point cloud augmentation method named Twin-DP3. Our approach leverages a 6-DoF pose estimation algorithm to track the manipulated object and uses a robot model to generate robot component point clouds in real time. We validate the effectiveness of this method through comprehensive experiments in both simulation and realworld environments, demonstrating notable improvements in policy robustness under novel viewpoints.