Mixed Reality-Based, Immersive, Semi-Autonomous Robotic Telemanipulation for the Execution of Peg-In-Hole Tasks

Abstract

Semi-autonomy in telemanipulation frameworks has the potential to reduce user cognitive load while preserving human perceptual oversight and decision-making capabilities. However, existing semi-autonomous telemanipulation systems are heavily dependent on calibration and hardware configurations, making rapid deployment difficult. Moreover, existing VR-based telemanipulation systems lack intuitive interaction mechanisms, requiring users to manage complex control interfaces. To address these limitations, we introduce an intuitive and immersive semi-autonomous robotic telemanipulation system that leverages a mixed reality (MR) headset with minimal hardware requirements. Requiring only CPU processing and coarse calibration procedures, the system combines human perception with autonomous control strategies through natural hand tracking and finger gestures to achieve precise, reliable task execution. To validate this approach, we conducted thorough evaluations involving complex peg-in-hole tasks and compared performance with and without the proposed control strategy. The results highlight that our system demonstrates robust performance, and the proposed control strategy further enhances its stability and effectiveness.