Safety Probability Estimation in Dimension Reduction Space for Model-Free Safe Reinforcement Learning of Robotics

Abstract

Reinforcement learning (RL) for robotics poses significant consideration of safety during training. However, a major challenge of safe reinforcement learning (SRL) methods is the curse of dimensionality. Although existing SRL methods utilizing dimensionality reduction (DR) approaches could provide safety probability estimation of a high-dimensional state according to a low dimensional safe region, their core safety estimators are not comprehensive. In this letter, we develop a novel, purely data-driven safety probability estimator which considers both the uncertainty of information loss caused by DR and the uncertainty of insufficient data caused by data-driven methods. This estimator does not need manual selection of parameters and the estimation is rather accurate even with a small amount of data. We theoretically prove that the estimator converges to the true safety probability. Existing RL algorithms (PPO, SAC) using this estimator can directly train control policies in real physical robots with a significant enhancement of training safety. The effectiveness of the algorithm are verified by conducting experiments for a quadruped robot in both practical and simulation environments, where a 34D-to-2D safety estimator is implemented to guarantee a 83% success rate of safe control policy.