Assembly Action Understanding from Fine-Grained Hand Motions, a Multi-camera and Deep Learning Approach
Enrique Coronado, Kosuke Fukuda, Ixchel G. Ramirez-Alpizar, Natsuki Yamanobe, Gentiane Venture, Kensuke Harada
Abstract
This article presents a novel software architecture enabling the analysis of assembly actions from fine-grained hand motions. Unlike previous works that compel humans to wear ad-hoc devices or visual markers in the human body, our approach enables users to move without additional burdens. Modules developed are able to: (i) reconstruct the 3D motions of body and hands keypoints using multi-camera systems; (ii) recognize objects manipulated by humans, and (iii) analyze the relationship between the human motions and the manipulated objects. We implement different solutions based on OpenPose and Mediapipe for body and hand keypoint detection. Additionally, we discuss the suitability of these solutions for enabling real-time data processing. We also propose a novel method using Long Short-Term Memory (LSTM) deep neural networks to analyze the relationship between the detected human motions and manipulated objects. Experimental validations show the superiority of the proposed approach against previous works based on Hidden Markov Models (HMMs).
BibTeX
@inproceedings{iros2021_assemblyactionun,
title = {Assembly Action Understanding from Fine-Grained Hand Motions, a Multi-camera and Deep Learning Approach},
author = {Enrique Coronado and Kosuke Fukuda and Ixchel G. Ramirez-Alpizar and Natsuki Yamanobe and Gentiane Venture and Kensuke Harada},
booktitle = {IROS 2021},
year = {2021}
}