A Non-Homogeneity Mapless Navigation Based on Hierarchical Safe Reinforcement Learning in Dynamic Complex Environments

Abstract

Addressing safe and efficient navigation in dynamic, realistic, and complex environments stands as a pivotal inquiry within the realm of robotics. Recently, numerous learning-based methods are introduced into the field of navigation, yielding notable outcomes. In this letter, we propose a hierarchical safe reinforcement learning navigation approach (HSRLN) for mapless navigation. It trains mapless navigation policies for non-homogeneous complex scenarios in a hierarchical manner through a kind of three-stage learning, global planning reinforcement learning (RL) + expert imitation learning (IL) + transfer RL (TRL). The innovations of this work are fourfold: a) It effectively reduces the difficulty of training for complex navigation by effectively narrowing the task horizon of RL through a hierarchical framework. b) We designed an imitation learning method based on Relative Driving Safety Index (RDSI) [1] to focus on learning critical expert actions. c) It employs a TRL approach to improve generalization under non-homogeneity assumptions by fine-tuning the policy. d) HSRLN extracts significant features important for navigation decisions from raw observations via velocity obstacle modeling. Experiments indicate that it has performs better than existing hierarchical RL navigation methods (HDRL [2], SRL-ORCA [3]). Relative to SRL-ORCA, it improves navigation success by 12.1% under the non-homogeneity assumption. Videos are available at https://youtu.be/24h9JmcIfMw.