Dynamic Legged Ball Manipulation on Rugged Terrains With Hierarchical Reinforcement Learning

Abstract

Achieving reliable object manipulation while traversing complex terrains is the missing link between agile quadruped locomotion and practical autonomy. Specifically, using traditional end-to-end reinforcement learning (RL) for dynamic ball manipulation in rugged environments presents two key challenges. The first is coordinating distinct motion modalities to integrate terrain traversal and ball control seamlessly. The second is overcoming sparse rewards in end-to-end RL, which impedes efficient policy convergence. To address these challenges, we propose a hierarchical RL framework. A high-level policy, informed by proprioceptive data and ball position, adaptively switches between pre-trained low-level skills such as ball dribbling and rough terrain navigation. We further propose Dynamic Skill-Focused Policy Optimization to suppress gradients from inactive skills and enhance critical skill learning. Both simulation and real-world experiments validate that our method outperforms baseline approaches in dynamic ball manipulation across rugged terrains, highlighting its effectiveness in challenging environments.