Fine-Grained Task Planning for Service Robots Based on Object Ontology Knowledge via Large Language Models

Abstract

In domestic environment, the successful execution of service tasks heavily relies on the robot's capability to identify and understand objects within its surrounding. This crucial process predominantly takes place during task planning, prior to the actual performance of service tasks. Therefore, it is vital that the robot is capable of formulating object-specific action sequences through task planning. In this letter, we propose the Fine-Grained Task Planning (FGTP) framework, an innovative method that combines object ontology knowledge with Large Language Models (LLMs) to create detailed action sequences. The FGTP framework is uniquely designed to process both text descriptions of service tasks and images of relevant objects, enabling a thorough comprehension of object attributes essential for task execution. Moreover, we have developed a set of rules based on these attributes to assist in the robot's decision-making process. In scenarios where service tasks fail because the object is in an unsuitable state, our framework deploys a logic-based reasoning method, concentrating on object attributes to identify suitable substitutes. This process leverages a pre-established semantic map to locate these alternatives, thus enabling a transition back to standard task planning. Our evaluations, conducted in both the VirtualHome simulation environment and with the TIAGo real robot, demonstrate the efficacy of our approach. This confirms our framework's capability to generate practical and implementable plans for various service tasks.