Self-Supervised Adaptive Transformer for Surgical Step Recognition in Robotic-Assisted Radical Prostatectomy

Abstract

The automatic recognition of surgical steps is essential for enhancing situational awareness and workflow automation in robotic-assisted surgery. However, existing vision-based approaches exhibit limitations in effectively leveraging rich spatial-temporal information from surgical videos, particularly when addressing generalization challenges across different clinical centers, surgeons, and patient populations. Current methods struggle with domain adaptation when deployed in diverse real-world settings due to substantial variations in surgical techniques, anatomical presentations, imaging conditions (such as lighting and white balance), and data preprocessing protocols. To address these limitations, we propose ProstaFormer (Prostatectomy Steps Transformer with Adaptive Feature Fusion), a framework that integrates a vision learner pre-trained via MAE with transformer-based temporal features through an adaptive fusion mechanism. Our approach intelligently combines spatial and temporal representations using position-aware attention weighting, enabling robust recognition of complex surgical workflow patterns across diverse clinical environments. Furthermore, we incorporate a diffusion-based Temporal Adaptation module to rectify domain-specific temporal order differences. We curate and annotate the comprehensive RPSteps dataset and conduct extensive experiments on both GraSP and RPSteps datasets. ProstaFormer consistently outperforms strong baselines, demonstrating improved generalization to different hospitals, surgeons, and patient populations, as well as superior robustness to image degradation. These results highlight the potential of adaptive feature fusion, temporal adaptation, and self-supervised visual pre-training for advancing intelligent robotic-assisted surgical workflow recognition.