Vision-based Teleoperation of Shadow Dexterous Hand using End-to-End Deep Neural Network
Shuang Li, Xiaojian Ma, Hongzhuo Liang, Michael Görner, Philipp Ruppel, Bin Fang, Fuchun Sun, Jianwei Zhang
Abstract
In this paper, we present TeachNet, a novel neural network architecture for intuitive and markerless vision-based teleoperation of dexterous robotic hands. Robot joint angles are directly generated from depth images of the human hand that produce visually similar robot hand poses in an end-to-end fashion. The special structure of TeachNet, combined with a consistency loss function, handles the differences in appearance and anatomy between human and robotic hands. A synchronized human-robot training set is generated from an existing dataset of labeled depth images of the human hand and simulated depth images of a robotic hand. The final training set includes 400K pairwise depth images and joint angles of a Shadow C6 robotic hand. The network evaluation results verify the superiority of TeachNet, especially regarding the high-precision condition. Imitation experiments and grasp tasks teleoperated by novice users demonstrate that TeachNet is more reliable and faster than the state-of-the-art vision-based teleoperation method.
BibTeX
@inproceedings{icra2019_visionbasedteleo,
title = {Vision-based Teleoperation of Shadow Dexterous Hand using End-to-End Deep Neural Network},
author = {Shuang Li and Xiaojian Ma and Hongzhuo Liang and Michael Görner and Philipp Ruppel and Bin Fang and Fuchun Sun and Jianwei Zhang},
booktitle = {ICRA 2019},
year = {2019}
}