LLM-Informed Iterative Planning for Object Search and Relocation in Indoor Environments

Abstract

The process of object search and relocation in an indoor environment, while intuitive for humans, remains a complex challenge for robots. Enabling robots to perform this task autonomously could have a substantial impact towards automation in both domestic and industrial settings. In this article, assuming a familiar environment, a set of target objects with their desired locations, and a robot with limited carrying capacity, we propose a novel methodology for object search and relocation. Given the human-like intuition exhibited by modern large language models (LLMs), they can be leveraged to guide object localization based on environmental context. Our approach integrates LLM-based prediction with graph-based path planning to create a human-like iterative search and relocation framework. The framework consists of an LLM predictor that suggests likely object locations (along with a likelihood score) and an adaptive path planner that dynamically updates the robot’s future path as new information becomes available during the search process. Prior relevant literature that employs LLM inference in indoor environments primarily focuses on assigning new or misplaced objects to appropriate locations. The aspect of enabling a search for a set of missing objects and planning their relocation to desired locations sets this article apart from prior literature. We compare our method to a patrol-based baseline with respect to the distance traversed by the robot in completing the search and relocation mission. In a medium sized indoor environment we demonstrate that it outperforms the baseline on an average by 31.2%.