LLM-Diffu: Robot Dexterous Grasp Generation Network With Diffusion Model and LLM

Abstract

Generating accurate dexterous grasps for objects with complex geometries remains a critical challenge in robotic manipulation. The paper proposes LLM-Diffu, a novel network architecture built on the principles of diffusion models. This architecture integrates a specialized basis point set (BPS) point cloud encoder, a denoising network, and large language models (LLMs) to generate feasible, diverse, and high-quality dexterous grasps. For the dexterous hand grasp dataset, the paper proposes a method for constructing the dataset, which incorporates a multimodal large language model (MLLM) to enhance the grasp quality of the dataset and is applicable to the construction of various types of datasets. Simulated experiments show our LLM-Diffu outperforms state-of-the-art methods, achieving 79.6% and 83.2% grasp success rates on DexGraspNet and MultiDex datasets, respectively. Our constructed dataset has strong generalizability, supporting training of various dexterous grasp generation methods with promising results. Finally, real-world robotic experiments confirm the practical applicability of our constructed dataset.