Climb With SHERPA: Heuristic-Guided Reinforcement Learning via Segmented Experience Relay

Abstract

In sparse-reward, long-horizon domains, reinforcement learning (RL) often suffers from slow convergence and instability, complicating robotic manipulation. Previous heuristic-guided approaches have relied on step-level actions and imitation loss, but struggle to maintain temporal coherence or solve multi-stage tasks. We present SHERPA (Segmented Heuristic Experience Relay for Policy Assistance), which alternates control between a heuristic policy and an RL policy in contiguous segments, preserving coherent sub-trajectories and yielding more stable learning. Unlike imitation-based designs, SHERPA learns entirely through RL optimization, enabling the policy to benefit from and surpass heuristic guidance. We also incorporate phase-specific rewards derived from heuristic rules to provide targeted subgoal feedback. Across ten tasks in the Fetch and Panda suites, including four long-horizon benchmarks, SHERPA consistently outperforms RL, IL, and heuristic-guided baselines and demonstrates robustness even under degraded heuristics. Real-world experiments on a UR5 robot further confirm SHERPA's practical scalability.