Mobile Manipulation Instruction Generation from Multiple Images with Automatic Metric Enhancement

Abstract

We consider the problem of generating mobile manipulation instructions based on a target object image and receptacle image. Conventional image captioning models are not able to generate appropriate instructions because their architectures are typically optimized for single-image. In this study, we propose a model that identifies both the target object and receptacle to generate free-form instruction sentences for mobile manipulation tasks. Furthermore, we introduce a novel training method, the human centric calibration phase that combines learning-based automatic evaluation metrics with n-gram based automatic evaluation metrics. This method enables the model to learn the co-occurrence relationships between words and appropriate paraphrases. The results demonstrate that our proposed method outperforms baseline methods including representative multimodal large language models on all automatic evaluation metrics. Moreover, physical experiments reveal that using our method to augment data on language instructions improves the performance of an existing multimodal language understanding model for mobile manipulation.