Precise Hand-Arm Teleoperation of Dexterous Robotic Manipulator via Markerless Vision and High Density sEMG Fusion
Jiahang Liu, Zhipeng Li, Renzhen Le, Xiao Zhang, Naohiko Sugita, Zhenzhi Ying, Liming Shu
Abstract
Teleoperation is crucial for robots in human-robot interaction and collaboration. It requires multi-scale human movement intention mapping in a natural and precise way to complete tasks. However, current human intention mapping methods for human-robot interface based only on vision or physiological information still face the challenge of incomplete information. In this letter, we have developed a novel hand-arm teleoperation system based on single-view markerless vision and high-density surface electromyography (HD-sEMG) fusion data, which can accurately estimate hand position, gestures, and grasp forces of human operators to achieve teleoperation of dexterous robotic manipulator. The hand tracking module obtained the positional and angular coordinates of the human hand based on RGB-depth (RGB-D) visual information as the end-effector pose servo input for the robotic arm. The fusion data of vision and EMG signals were input to a gesture classification network and a force regression network to estimate human hand gestures and grasp forces. Teleoperation experiment tasks were conducted on ten subjects to test our proposed system under various conditions. We compared the trajectories between the human hand and the robot hand and did the quantitative evaluation on the performance of fusion features-based gesture recognition and force regression. The results have validated the practicality and robustness of the teleoperation system.
BibTeX
@inproceedings{ral2026_precisehandarmte,
title = {Precise Hand-Arm Teleoperation of Dexterous Robotic Manipulator via Markerless Vision and High Density sEMG Fusion},
author = {Jiahang Liu and Zhipeng Li and Renzhen Le and Xiao Zhang and Naohiko Sugita and Zhenzhi Ying and Liming Shu},
booktitle = {RA-L 2026},
year = {2026}
}