Deep Learning–Driven Tumor Boundary Estimation Using Robotic Palpation in Minimally Invasive Surgery

Abstract

Accurate estimation of tumor boundaries is critical for ensuring adequate surgical margins in robot-assisted minimally invasive surgery (RMIS). In this study, we present a method that estimates tumor boundaries in RMIS using sweeping palpation data acquired with a single force/torque (F/T) sensor. From the reconstructed surface, tissue displacement and normal force were derived to calculate stiffness, which was then used to construct a stiffness map. To reduce noise and enhance feature representations, we employed a sparse autoencoder (SAE). The SAE outputs were subsequently clustered with a Gaussian mixture model (GMM) and K-means to segment the tumor from normal tissue. Experiments with phantom models and an ex vivo model demonstrated that the SAE-based approach significantly improved the Dice similarity coefficient (DSC) and sensitivity while maintaining specificity, and reduced the Hausdorff distance (HD and HD95) and average symmetric surface distance (ASSD), compared with results from raw data. Importantly, when evaluated under clinically relevant surgical margin conditions, the estimated HD consistently remained below threshold across all models. These results indicate that the proposed method achieves both high accuracy and clinical feasibility without additional imaging devices or displacement sensors, highlighting its potential to support margin minimization and organ function preservation in RMIS.