Sample-Efficient Learning of a Robotic Tube Insertion Policy

Abstract

Reinforcement learning for manipulation with real robots is complicated when faced with object variations and contact dynamics.Further, achieving support across a category of objects for manipulation is in itself a challenging task. A recent work that addresses training inefficiencies on a real robot is the Contrastive Pre-training and Data Augmentation for Efficient Robotic Learning (CoDER) framework. It uses a combination of contrastive-pretraining, data augmentation, expert demonstrations and reinforcement learning to achieve sample-efficiency. To improve sample-efficiency further, we used force/torque inputs, pre-trained image encoders, layer normalization, and we warmed up our actor-critic networks before training. In this work, we propose these improvements to the original CoDER for stable and efficient training of a real robot for fine manipulation - we insert medical tubes into a rack. Our policy is trained on a single tube and it is able to support different tubes in the test set. The experimental results show an average insertion success rate of 98.1% across several different types of medical tubes. We achieve this robustness with a policy trained in only 800 episodes over approximately 6 hours using sparse rewards.