Language-Guided Hierarchical Planning with Scene Graphs for Tabletop Object Rearrangement

Abstract

Spatial relationships between objects are key to achieving well-arranged scenes. In this paper, we address the robotic rearrangement task by leveraging these relationships to reach configurations that are both well-arranged and satisfying the given language goal. We propose a hierarchical planning framework that bridges the gap between abstract language inputs and concrete robotic actions. A scene graph is central to this approach, serving as both an intermediate representation and the state for high-level planning, capturing the relationships among objects effectively and reducing planning complexity. This also enables the proposed method to handle more general language goals. To achieve this, we leverage a large language model (LLM) to convert language goals into a scene graph, which becomes the goal for high-level planning. In high-level planning, we plan transitions from the current scene graph to the goal scene graph. To integrate high-level and low-level planning, we introduce a network that generates a physical configuration of objects from a scene graph. Low-level planning then verifies the high-level plan’s feasibility, ensuring it can be executed through robotic manipulation. Through experiments, we show that the proposed method handles general language goals effectively and produces human-preferred rearrangements compared to other approaches, demonstrating its applicability on real robots without requiring sim-to-real adjustments.