Language-Embedded 6D Pose Estimation for Tool Manipulation

Abstract

Robotic tool manipulation requires understanding task-relevant semantics under visually challenging conditions, such as shape variation and occlusion. This paper presents a novel framework for Language-Embedded Semantic 6D Pose Estimation that combines natural language instructions with 3D point cloud data to achieve category-level 6D pose estimation of tools' functional parts. By embedding semantic information from large language models (LLMs) and leveraging a diffusion-based pose estimator, our approach achieves robust generalization across diverse tool categories. We introduce a comprehensive synthetic dataset, tailored for tool manipulation scenarios, with annotated 6D poses of functional parts. Extensive experiments conducted on both the synthetic dataset and real-world robots demonstrate our system's ability to interpret natural language commands, predict poses of functional parts, and perform manipulation tasks with significant improvements in accuracy and generalization.