Image-to-Force Estimation for Soft Tissue Interaction in Robotic-Assisted Surgery Using Structured Light

Abstract

For Minimally Invasive Surgical (MIS) robots, accurate haptic interaction force feedback is essential for ensuring the safety of interacting with soft tissue. However, the majority of existing MIS robotic systems cannot facilitate direct measurement of the interaction force with hardware sensors due to space limitations. This letter introduces an effective vision-based scheme that utilizes a One-Shot structured light projection with a designed pattern on soft tissue coupled with haptic information processing through a trained image-to-force neural network. The images captured from the endoscopic stereo camera are analyzed to reconstruct high-resolution 3D point clouds for soft tissue deformation. The proposed methodology involves a modified PointNet-based force estimation method, which has demonstrated proficiency in accurately representing the intricate mechanical properties of soft tissue. To validate the efficacy of the proposed methodology, numerical force interaction experiments were conducted on three silicon materials with varying stiffness levels. The experimental results substantiate the efficacy of the proposed methodology.