SemSegGrasp: Plug-and-play Task-oriented Grasping via Semantic Segmentation

Abstract

Task-oriented grasping (TOG) involves grasping specific parts of an object based on a task instruction. Existing methods generally integrate semantic analysis with grasp detection, resulting in low efficiency and poor generalization when dealing with new tasks and hardware. To address these problems, we propose SemSegGrasp, which reformulates TOG as a semantic segmentation problem based on point cloud and text matching. By decomposing TOG into semantic segmentation and grasp detection, SemSegGrasp can significantly enhance both the efficiency and generalization performance of TOG. Moreover, it can be combined with any off-the-shelf grasp detection algorithms in a plug-and-play manner. For semantic segmentation, SemSegGrasp first utilizes a Vision-Language Model (VLM) to generate local geometric descriptions of the target object. These descriptions are then fed into a Large Language Model (LLM) along with user instructions to obtain operational guidance. Subsequently, we separately encode the input point cloud and the operational guidance and obtain their features. Leveraging multi-head cross-attention, we conduct a matching process between these two types of features to predict the probability of each point serving as a TOG grasp point, i.e. semantic segmentation. Finally, the grasp pose is determined by fusing the segmentation results with the candidate poses generated by an existing grasp detection algorithm. Experimental results on the publicly available TaskGrasp dataset and a real-world setting show that our SemSegGrasp method achieves state-of-the-art performance, outperforming existing methods by at least 5% and 10% on new tasks, respectively.