Lyapunov Constrained Safe Reinforcement Learning for Multicopter Visual Servoing

Abstract

Traditional methods based on Lyapunov analysis and learning-based approaches such as reinforcement learning (RL) are two powerful tools in visual servo tasks. Traditional methods are interpretable and their stability can be guar-anteed by Lyapunov analysis. However, they tend to have a high dependency on an accurate system dynamic model. RL approaches learn to act based on past experiences and thus have higher adaptability on disturbances and errors. However, the training process is long and its safety or stability is generally hard to guarantee, making real-world training risky. In this paper, we propose a residual RL framework for training a multicopter to finish visual servo tasks under disturbances, guided by the system safety in terms of system Lyapunov function. Such an approach compensates for the lack of disturbance-rejection ability of the traditional method, and optimizes stability explicitly so the RL agent makes safer actions both during the training and in the final policy. A comparison between our approach and the baselines is provided in simulation, and real-world experiments on a multicopter are also carried out to show our effectiveness. We believe that this work moves one step toward achieving RL applications on real-world robotic systems.