Stepping Locomotion for a Walking Excavator Robot using Hierarchical Reinforcement Learning and Action Masking

Abstract

The employment of walking excavator robots, endowed with hybrid locomotion capabilities, holds considerable promise in facilitating the execution of intricate tasks in challenging terrain environments. A critical skill for such a system pertains to traversing obstacles through stepping locomotion, a process entailing the momentary disengagement of the end-effectors from the ground. Existing solutions are encumbered by two significant limitations. Primarily, they are often too cumbersome to develop and implement due to the complexity of the problem formulations. Secondly, they present restrictions on the available avenues to influence the behavior, hindering the effective leveraging of domain knowledge to achieve the intended objective.This research proposes an alternative approach to learning the stepping locomotion. The proposed method employs a hierarchical reinforcement learning strategy, wherein the complex control task is decomposed into multiple subtasks, each aligned with a sub-objective defined as a reward function. The training of these subtasks is conducted individually, starting from the lowest level and progressing to the higher levels, predominantly utilizing deep reinforcement learning. Additionally, the masking of invalid actions is utilized to guide the controller during training, offering enhanced opportunities to influence behavior while using only simple formulations. Notably, the proposed approach has been successfully trained for three distinct stepping scenarios: obstacle, step, and gap, underscoring the versatility of the controller. The design of the controller, along with the results of training and evaluation in simulation, is presented herein.