A Mobile Robot Hand-Arm Teleoperation System by Vision and IMU
Shuang Li, Jiaxi Jiang, Philipp Ruppel, Hongzhuo Liang, Xiaojian Ma, Norman Hendrich, Fuchun Sun, Jianwei Zhang
Abstract
In this paper, we present a multimodal mobile teleoperation system that consists of a novel vision-based hand pose regression network (Transteleop) and an IMU (inertial measurement units)-based arm tracking method. Transteleop observes the human hand through a low-cost depth camera and generates not only joint angles but also depth images of paired robot hand poses through an image-to-image translation process. A keypoint-based reconstruction loss explores the resemblance in appearance and anatomy between human and robotic hands and enriches the local features of reconstructed images. A wearable camera holder enables simultaneous hand-arm control and facilitates the mobility of the whole teleoperation system. Network evaluation results on a test dataset and a variety of complex manipulation tasks that go beyond simple pick-and-place operations show the efficiency and stability of our multimodal teleoperation system.
BibTeX
@inproceedings{iros2020_amobilerobothand,
title = {A Mobile Robot Hand-Arm Teleoperation System by Vision and IMU},
author = {Shuang Li and Jiaxi Jiang and Philipp Ruppel and Hongzhuo Liang and Xiaojian Ma and Norman Hendrich and Fuchun Sun and Jianwei Zhang},
booktitle = {IROS 2020},
year = {2020}
}