Surgical Workflow Prediction via Visual Information in Laparoscopic and Robot-Assisted Surgery

Abstract

Surgical workflow prediction is critical for enhancing safety and providing real-time guidance in Computer-Assisted Surgery (CAS), particularly in laparoscopic and Robot-Assisted Surgery (RAS). We propose a novel visual information-based method for predicting surgical workflows at fine temporal scales. Our model adopts a two-stage training strategy. First, a ResNet50 backbone is trained to extract robust spatial features from individual surgical frames. Then, a predictive network is trained on sequential features. This network integrates a novel Temporal Feature Recorder (TFR) module, which aggregates both local and global temporal information from observed frame sequences, together with a Cross-Attention mechanism that fuses these temporal features with spatial information. A Temporal Convolutional Network (TCN) subsequently leverages this aggregated spatiotemporal representation to predict future feature states and corresponding workflow classifications. We evaluate our method on the JIGSAWS (activity level) and Cholec80 (phase level) datasets. It achieves superior classification accuracy (60.37%), significantly outperforming baselines on JIGSAWS, and delivers competitive accuracy (85.73%) with the lowest prediction error (MAE: 0.3672, MSE: 0.2148) on Cholec80 compared to existing baselines.