Region-Driven Adaptation for Constrained Reinforcement Learning-Based Control

Abstract

This paper presents a region-driven adaptation for constrained reinforcement learning (RL)-based control for systems with unknown nonlinear dynamics. For safety-critical control, several noteworthy results have been presented that ensure safety through the control barrier function (CBF) theory. One of the challenges for such control designs lies in the feasibility of a solution for the quadratic program (QP). The performance of CBF-based QP is influenced by any fixed parameters of QP, potentially leading to overly conservative behavior or safety violations. Firstly, we introduce a region-based adaptation of class-K function α(.) as an adaptive CBF parameter to avoid overly conservative behavior or safety violations. Then, we incorporate the adaptive CBF parameter into the RL algorithm and present an adaptively constrained proximal policy optimization algorithm (α-CPPO). The algorithm not only achieves the desired performance but also makes RL aware of its state with relation to safe behavior through α(.). The safe RL framework also works in scenarios where the CBF-based QP is infeasible, enabling the system to recover. Finally, we illustrate the effectiveness of the presented approach for an energy industry application, namely an ESP-lifted well.