Visual Proprioception and Contact Prediction for Soft Robotic Arms Using Onboard Sensing and Generative Models

Abstract

Soft continuum robots possess inherent compliance and ensure safe interactions. Whilst advantageous, their high degrees of freedom pose significant challenges for estimating the robot's configuration and sensing external contacts. Existing analytical methods rely on finite state-representations that do not accurately capture the complex deformations in these systems. Learning-based approaches have shown promise, as they do not require any prior assumptions about the state representation; however, there are still challenges in extending them to handle external contacts. To address these challenges, this study proposes a novel perception framework for soft robotic arms by using onboard sensor data and generative models to develop image-based perception systems capable of reconstructing the arm's state, even under external contact. Our approach features an automated data preprocessing pipeline that isolates the robot from the environmental background, enabling the training of robust perception models. It then utilizes onboard IMU data and actuation commands to generate a high-dimensional visual representation of the robot's state, while also detecting contact locations with high accuracy.