Recognizing Real-World Intentions using A Multimodal Deep Learning Approach with Spatial-Temporal Graph Convolutional Networks
Jiaqi Shi, Chaoran Liu, Carlos Toshinori Ishi, Bowen Wu, Hiroshi Ishiguro
Abstract
Identifying intentions is a critical task for comprehending the actions of others, anticipating their future behavior, and making informed decisions. However, it is challenging to recognize intentions due to the uncertainty of future human activities and the complex influence factors. In this work, we explore the method of recognizing intentions alluded under human behaviors in the real world, aiming to boost intelligent systems' ability to recognize potential intentions and understand human behaviors. We collect data containing real-world human behaviors before using a hand dispenser and a temperature scanner at the building entrance. These data are processed and labeled into intention categories. A questionnaire is conducted to survey the human ability in inferring the intentions of others. Skeleton data and image features are extracted inspired by the answer to the questionnaire. For skeleton-based intention recognition, we propose a spatial-temporal graph convolutional network that performs graph convolutions on both part-based graphs and adaptive graphs, which achieves the best performance compared with baseline models in the same task. A deep-learning-based method using multimodal features is proposed to automatically infer intentions, which is demonstrated to accurately predict intentions based on past behaviors in the experiment, significantly outperforming humans.
BibTeX
@inproceedings{iros2023_recognizingrealw,
title = {Recognizing Real-World Intentions using A Multimodal Deep Learning Approach with Spatial-Temporal Graph Convolutional Networks},
author = {Jiaqi Shi and Chaoran Liu and Carlos Toshinori Ishi and Bowen Wu and Hiroshi Ishiguro},
booktitle = {IROS 2023},
year = {2023}
}