A Hybrid Framework Using Diffusion Policy and Residual RL for Force-Sensitive Robotic Manipulation

Abstract

Force-sensitive manipulation is essential for tasks such as cleaning, polishing, and surgical assistance, yet it remains challenging due to complex contact dynamics and the need for real-time adaptation. We propose DP-RRL, a hybrid learning framework that combines a diffusion policy (DP) for imitation learning (IL) with a residual reinforcement learning (RL) module to tackle these challenges. First, the DP leverages expert demonstrations to generate smooth, expert-like motion-force trajectories. Next, a residual RL component refines these trajectories online by adapting force commands to account for unmodeled contact dynamics. Our approach incorporates multimodal perception, including RGB point clouds, force feedback, and proprioceptive data, and achieves robust force-sensitive manipulation in unstructured environments. We validate DP-RRL on a 7-DOF robotic arm performing basin-cleaning tasks, where it demonstrates an average success rate of 88% across 12 unseen scenarios, outperforming state-of-the-art baselines.