Interactively Picking Real-World Objects with Unconstrained Spoken Language Instructions
Jun Hatori, Yuta Kikuchi, Sosuke Kobayashi, Kuniyuki Takahashi, Yuta Tsuboi, Yuya Unno, Wilson Ko, Jethro Tan
Abstract
Comprehension of spoken natural language is an essential skill for robots to communicate with humans effectively. However, handling unconstrained spoken instructions is challenging due to (1) complex structures and the wide variety of expressions used in spoken language, and (2) inherent ambiguity of human instructions. In this paper, we propose the first comprehensive system for controlling robots with unconstrained spoken language, which is able to effectively resolve ambiguity in spoken instructions. Specifically, we integrate deep learning-based object detection together with natural language processing technologies to handle unconstrained spoken instructions, and propose a method for robots to resolve instruction ambiguity through dialogue. Through our experiments on both a simulated environment as well as a physical industrial robot arm, we demonstrate the ability of our system to understand natural instructions from human operators effectively, and show how higher success rates of the object picking task can be achieved through an interactive clarification process.
BibTeX
@inproceedings{icra2018_interactivelypic,
title = {Interactively Picking Real-World Objects with Unconstrained Spoken Language Instructions},
author = {Jun Hatori and Yuta Kikuchi and Sosuke Kobayashi and Kuniyuki Takahashi and Yuta Tsuboi and Yuya Unno and Wilson Ko and Jethro Tan},
booktitle = {ICRA 2018},
year = {2018}
}