Trajectory-based Split Hindsight Reverse Curriculum Learning

Abstract

Grasping is one of the most fundamental problems in robotic manipulation. In recent years, with the development of data-driven methods, reinforcement learning has been used in solving robotic grasping problems. However, grasping is a long-horizon and sparse reward task, whose natural reward only appears when the task is successfully achieved. Therefore, it brings great challenges to the deployment of reinforcement learning methods. To tackle this difficulty, we propose a new method called Trajectory-based Split Hindsight Reverse Curriculum Learning. This method of reverse learning from the goal can greatly improve the learning efficiency and the final performance of the tasks. Specifically, based on referred trajectories, the agent starts to learn in a small state space near the goal and then gradually in larger state spaces until covering the entire state space. Through split hindsight experience replay, the sampled trajectory is divided into segments that match the current subspace's size; then, they are modified to successful trajectories to enable more efficient learning. In both simulation and real-world experiments, our method surpasses the existing methods and achieves the goal-oriented grasping tasks with higher success rates and better data efficiencies. The detailed experimental results can be viewed at https://youtu.be/7uNRzmRZhDk.