Graph2Scene: Versatile 3D Indoor Scene Generation with Interaction-aware Scene Graph

Abstract

Embodied artificial intelligence requires a wide variety of large-scale simulated environments for development. Previous scene reconstruction approaches based on multiview images can produce high-fidelity 3D scenes but lack diversity. In contrast, existing prompt-based scene generation approaches can produce diverse scenes but lack fine-grained controls. To bridge these two fields, the scene graph provides the key relationships within a scene, while offering flexible controls. However, 3D scene generation from scene graphs is challenging and under-explored. In this paper, we propose a scene graph-based 3D indoor scene generation method for the efficient simulated environment creation, that maintains both high diversity and fine-grained control. Specifically, we first introduce an interaction-aware scene graph to merge object nodes with hierarchical interaction relationships, which alleviates levitation and interference issues during the scene generation. Then, we employ the large language model (LLM) for instruction-driven 3D layout generation with carefully designed prompts. Finally, a 3D large generation model is utilized to generate the content for each node of the interaction-aware scene graph, which is then transformed based on the corresponding bounding box in the 3D layout. The experiments demonstrate that the proposed method achieves state-of-the-art performance on 3D indoor scene generation. Additionally, the proposed method exhibits fine-grained controls at the object level, while providing a high diversity of layouts, geometry, and textures.