PhysiXDeform: Real-Time Vision-Guided Soft-Tissue Deformation Prediction With Physical Priors

Abstract

Accurate prediction of soft tissue deformation from endoscopic video is critical for robot-assisted, image-guided interventions. However, it remains challenging due to occlusions, complex dynamics, and the absence of direct physical measurements. Existing vision-based approaches often impose rigid or simplified motion priors, limiting their generalisability and physical interpretability. In this paper, we introduce a vision-guided, physics-informed framework that predicts multi-step 3D deformation by integrating stereo disparity, optical flow, and tracked keypoints into a spatio-temporal graph. Motion features, including velocity, acceleration, and directional strain, are encoded as node and edge attributes and processed via a graph network. The model is trained end-to-end with self-supervised objectives promoting geometric alignment, temporal coherence, and motion consistency. Across synthetic and real surgical datasets, our method achieves up to <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">28.8%</b> reduction in Chamfer Distance (CD) at short prediction horizons, and an average <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">43.9%</b> reduction over extended-horizon forecasts. In cross-scene adaptation, our model achieves average CD reductions of <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">12.9%</b> with light tuning and <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">12.2%</b> with heavy tuning, where the latter shows slight overfitting but still outperforms mesh-based baselines and supports real-time operation.