Visually Grounding Language Instruction for History-Dependent Manipulation
Hyemin Ahn, Obin Kwon, Kyungdo Kim, Jaeyeon Jeong, Howoong Jun, Hongjung Lee, Dongheui Lee, Songhwai Oh
Abstract
This paper emphasizes the importance of a robot's ability to refer to its task history, especially when it exe-cutes a series of pick-and-place manipulations by following language instructions given one by one. The advantage of referring to the manipulation history can be categorized into two folds: (1) the language instructions omitting details but using expressions referring to the past can be interpreted, and (2) the visual information of objects occluded by previous manipulations can be inferred. For this, we introduce a history-dependent manipulation task which objective is to visually ground a series of language instructions for proper pick-and-place manipulations by referring to the past. We also suggest a relevant dataset and model which can be a baseline, and show that our model trained with the proposed dataset can also be applied to the real world based on the CycleGAN. Our dataset and code are publicly available on the project website: https://sites.google.com/view/history-dependent-manipulation.
BibTeX
@inproceedings{icra2022_visuallygroundin,
title = {Visually Grounding Language Instruction for History-Dependent Manipulation},
author = {Hyemin Ahn and Obin Kwon and Kyungdo Kim and Jaeyeon Jeong and Howoong Jun and Hongjung Lee and Dongheui Lee and Songhwai Oh},
booktitle = {ICRA 2022},
year = {2022}
}