Safe Policy Optimization With Cost Practical Stability: A DQ-Learning Method

Abstract

Safety is a critical requirement in robotics, leading to extensive research in Safe Reinforcement Learning. While most existing methods utilize soft constraints to limit cumulative violations, robotic control scenarios demand that instantaneous safety costs also converge rapidly following a safety excursion. Inspired by practical stability from safety control, we formulate cost practical stability as a novel constraint. We introduce DQ-learning, an algorithm derived from D-learning, a differential predictive learning method ensuring system stability, to satisfy this constraint. DQ-learning merges the safety stability of D-learning with the reward optimality of Q-learning, incorporating this constraint within a policy optimization framework. Evaluations on multiple robotic safety benchmarks and a sim-to-real drone target-catching task demonstrate that DQ-learning significantly outperforms baseline methods in both constraint satisfaction and sample efficiency.