Multimodal Deformation Estimation of Soft Pneumatic Gripper During Operation

Abstract

Soft pneumatic robots are gaining significant attention due to their compliance and adaptability in unstructured environments. While emerging dual-chamber soft pneumatic robots can achieve complex 3D deformations beyond conventional single-axis bending, real-time proprioception remains challenging due to the high degrees of freedom and the complex interaction between chambers. To address this issue, we propose a multimodal learning-based sensing method that combines camera and inertial measurement unit (IMU) and then extracts full-body shape information using deep learning algorithms. Our method enhances proprioception by effectively processing high-dimensional sensor data, providing real-time feedback on the gripper shape. The average error of key points was found to be 3.67mm (Var 8.39) for our method, while the error was 4.36mm (Var 10.47) when a camera was used alone, or 9.32mm (Var 21.29) when an IMU was used alone. Our multimodal learning-based shape estimation and reconstruction empower soft pneumatic grippers to be seamlessly integrated into the embodied AI framework, significantly improving their reliability and thus paving the way for applications in service robotics, ehabilitation robotics, and human-robot collaborations.