DiffDef: A Diffusion Model for Generating Multimodal Goal Shapes from Demonstrations for Deformable Object Manipulation

Abstract

Deformable object manipulation is pivotal to numerous real-world robotic applications. A promising paradigm in this field is the shape servoing task, focusing on controlling deformable objects into desired goal shapes. However, prior works typically rely on impractical goal shape acquisition methods, such as laborious domain-knowledge engineering or manual manipulation. Crucially, existing methods fail in multi-modal goal settings, where multiple distinct goal shapes can all lead to successful task completion, a common scenario in many robotic applications. In this paper, we address this problem by developing DiffDef, a novel neural network that leverages a denoising diffusion model to learn a distribution over multiple valid goal shapes, rather than predicting a single deterministic outcome. DiffDef enables the generation of diverse goal shapes, thereby avoiding the mode-averaging artifacts inherent in deterministic models used by previous approaches. We demonstrate our method’s effectiveness on several robotic tasks inspired by both manufacturing and surgical applications, both in simulation and on two physical robotic platforms: the da Vinci Research Kit (dVRK) robot and a bimanual KUKA-based robotic system.