A Safety-Adjusted Policy Optimization Algorithm and Application for Obstacle Avoidance in the Quadcopter

Abstract

Ensuring the safety of various real-world applications based on reinforcement learning (RL), such as quadcopter control, robotic manipulators, and autonomous robots, remains a critical challenge, despite RL’s remarkable success in solving complex decision-making tasks. Existing on-policy Lagrangian optimization methods in safe RL typically use a single policy to balance the trade-off between safety and return without taking the potential benefits of adopting multiple policies into account. In this paper, a new on-policy method is proposed, named Safe-Adjusted Policy Optimization(SAPO), which is a dual-policy framework designed to address safety constraint violations in RL. By incorporating a cost-oriented policy to dynamically adjust a reward-oriented policy, the SAPO effectively resolves the trade-off between safety and return. Moreover, to enhance performance in carrying out high-dimensional tasks, the Kullback-Leibler (KL) divergence and a Gaussian kernel are employed in the distance functions to facilitate the training. In addition, a quadcopter-safe-navigation task is designed to overcome the drawback of previous research on quadcopter-safe-navigation with RL that only pays attention to reward function design without considering policy-level optimization. Finally, experimental results verify the feasibility of the designed task. Meanwhile, indicated by the test on real device, the proposed algorithm is easy to be implemented, offers performance guarantees, and outperforms existing safe RL baselines.