Semi-Supervised Language-Conditioned Grasping With Curriculum-Scheduled Augmentation and Geometric Consistency

Abstract

Language-Conditioned Grasping (LCG) is an essential skill for robotic manipulation and has attracted increasing interest. Recent LCG models have made great progress, but need numerous paired image-text-pose annotations for fully supervised learning, which are tedious and expensive. Semi-supervised learning has provided a viable solution, while they still encounter the following challenges for LCG: (i) Over-distorted data perturbations result in slow and unstable convergence for multi-modal inputs in the early stage. (ii) Inconsistency between the perceptive and grasping locations leads to a degradation of grasp accuracy. In this letter, we propose a semi-supervised language-conditioned grasping framework that achieves data-efficient object grounding and grasping detection based on language description. Concretely, we introduce a Curriculum-Scheduled augmentation and Geometric Consistency (CSGC) strategy to address the above problems. Concretely, We design a curriculum-scheduled augmentation to progressively improve data diversity from easy to difficult, facilitating stable knowledge distillation and model convergence. Meanwhile, we present a geometry-aware consistency regularization to constrain the region alignment between object perception and grasping confidence, improving the quality of pseudo-labels and grasp accuracy. Extensive experimental results demonstrate the effectiveness and practicability of our proposed method in the limited labeled data.