Safeguarded Progress in Reinforcement Learning: Safe Bayesian Exploration for Control Policy Synthesis
Rohan Mitta, Hosein Hasanbeig, Jun Wang, Daniel Kroening, Yiannis Kantaros, Alessandro Abate
Abstract
This paper addresses the problem of maintaining safety during training in Reinforcement Learning (RL), such that the safety constraint violations are bounded at any point during learning. As enforcing safety during training might severely limit the agent’s exploration, we propose here a new architecture that handles the trade-off between efficient progress and safety during exploration. As the exploration progresses, we update via Bayesian inference Dirichlet-Categorical models of the transition probabilities of the Markov decision process that describes the environment dynamics. We then propose a way to approximate moments of belief about the risk associated to the action selection policy. We demonstrate that this approach can be easily interleaved with RL and we present experimental results to showcase the performance of the overall architecture.
BibTeX
@article{Mitta_Hasanbeig_Wang_Kroening_Kantaros_Abate_2024, title={Safeguarded Progress in Reinforcement Learning: Safe Bayesian Exploration for Control Policy Synthesis}, volume={38}, url={https://ojs.aaai.org/index.php/AAAI/article/view/30137}, DOI={10.1609/aaai.v38i19.30137}, abstractNote={This paper addresses the problem of maintaining safety during training in Reinforcement Learning (RL), such that the safety constraint violations are bounded at any point during learning. As enforcing safety during training might severely limit the agent’s exploration, we propose here a new architecture that handles the trade-off between efficient progress and safety during exploration. As the exploration progresses, we update via Bayesian inference Dirichlet-Categorical models of the transition probabilities of the Markov decision process that describes the environment dynamics. We then propose a way to approximate moments of belief about the risk associated to the action selection policy. We demonstrate that this approach can be easily interleaved with RL and we present experimental results to showcase the performance of the overall architecture.}, number={19}, journal={Proceedings of the AAAI Conference on Artificial Intelligence}, author={Mitta, Rohan and Hasanbeig, Hosein and Wang, Jun and Kroening, Daniel and Kantaros, Yiannis and Abate, Alessandro}, year={2024}, month={Mar.}, pages={21412-21419} }