Multimodal Autonomous Robotic Long-Horizon Task Planning via Embodied Language Model and Behavior Trees

Abstract

Enabling robotic systems to perform long-horizon manipulation planning in real-world environments based on multimodal embodied perception and comprehension remains a longstanding challenge. Recent advancements in large language models (LLMs) have spurred the development of LLM-based planners; however, these approaches often rely on human-provided textual representations or extensive prompt engineering, lacking the ability to quantitatively interpret the environment. To overcome these limitations, we propose a novel framework that leverages LLMs and vision-language models (VLMs) to perform abstract reasoning and extract task-relevant representations from the environment using grounding mechanisms. To further enhance robotic capabilities, we introduce a systematic approach to constructing robotic skill libraries, enabling efficient generation of feasible and optimal actions. Unlike prior work, our LLM-based task planner reformulates user instructions into Planning Domain Description Language (PDDL) problems and employs Behavior Trees to represent the hierarchical structure of tasks, offering interpretable and modular task execution. Extensive evaluations on diverse real-world long-horizon manipulation tasks demonstrate the effectiveness of the proposed method, achieving an average success rate exceeding 80%. Furthermore, the framework functions as a high-level planner, empowering robots with substantial autonomy in unstructured environments by leveraging multimodal sensor inputs.