End-to-End Diffusion-Based 3D Object Reconstruction From Robotic Tactile Sensing

Abstract

Tactile sensing is essential for robotic perception in scenarios where visual input is limited or unavailable. In this work, we propose a fully tactile-based 3D object reconstruction framework that recovers object shapes exclusively from contact observations. A robotic system comprising a robotic arm, a robotic hand, and tactile sensors captures high-resolution tactile images during multi-contact grasps. These images are processed using TouchVIT to extract depth features, which are mapped to local point clouds via a CNN-based encoder and registered in 3D space using forward kinematics. To reconstruct the global object shape, we introduce TouchDiffusion, a conditional diffusion model that iteratively denoises Gaussian noise guided by the observed tactile geometry. We further develop a simulation environment built on TACTO and TactileGym to support data generation and enable sim-to-real transfer. Our framework enables high-fidelity, vision-free reconstruction of complex 3D objects from sparse tactile inputs. Experiments in both simulated and real-world settings validate the effectiveness and generalizability of our method, outperforming existing tactile-only approaches.