VDTF-ACT: ACT-based Multimodal Space Fine Manipulation Method with Visual Depth Tactile Fusion

Abstract

Autonomous fine manipulation in space for orbital assembly continues to present a critical challenge in the field of aerospace engineering. Under low-gravity conditions, during satellite manipulator operations on free-floating objects, the absence of significant gravitational forces and friction constraints leads to unpredictable relative motions between the manipulator’s end-effectors and objects, degrading the manipulation performance. This study proposes a fine manipulation method for satellite robots with floating platforms, grounded in multimodal perception and an enhanced Action Chunking with Transformer (ACT) architecture that enables multimodal state interaction. By integrating visual, tactile, and depth sensory data, the satellite robot’s space fine manipulation capabilities are substantially improved. From the experimental results generated from simulated environments, the proposed method achieves a success rate exceeding 80% for peg-in-socket insertion tasks, outperforming conventional approaches with a success rate of approximately 45%. Project Website: https://github.com/LSY0528/VDTF-ACT.