Learning to Shape Rewards Using a Game of Two Partners
David Mguni, Taher Jafferjee, Jianhong Wang, Nicolas Perez-Nieves, Wenbin Song, Feifei Tong, Matthew Taylor, Tianpei Yang
Abstract
Reward shaping (RS) is a powerful method in reinforcement learning (RL) for overcoming the problem of sparse or uninformative rewards. However, RS typically relies on manually engineered shaping-reward functions whose construc- tion is time-consuming and error-prone. It also requires domain knowledge which runs contrary to the goal of autonomous learning. We introduce Reinforcement Learning Optimising Shaping Algorithm (ROSA), an automated reward shaping framework in which the shaping-reward function is constructed in a Markov game between two agents. A reward-shaping agent (Shaper) uses switching controls to determine which states to add shaping rewards for more efficient learning while the other agent (Controller) learns the optimal policy for the task using these shaped rewards. We prove that ROSA, which adopts existing RL algorithms, learns to construct a shaping-reward function that is beneficial to the task thus ensuring efficient convergence to high performance policies. We demonstrate ROSA’s properties in three didactic experiments and show its superior performance against state-of-the-art RS algorithms in challenging sparse reward environments.
BibTeX
@article{Mguni_Jafferjee_Wang_Perez-Nieves_Song_Tong_Taylor_Yang_Dai_Chen_Zhu_Shao_Wang_Yang_2023, title={Learning to Shape Rewards Using a Game of Two Partners}, volume={37}, url={https://ojs.aaai.org/index.php/AAAI/article/view/26371}, DOI={10.1609/aaai.v37i10.26371}, abstractNote={Reward shaping (RS) is a powerful method in reinforcement learning (RL) for overcoming the problem of sparse or uninformative rewards. However, RS typically relies on manually engineered shaping-reward functions whose construc- tion is time-consuming and error-prone. It also requires domain knowledge which runs contrary to the goal of autonomous learning. We introduce Reinforcement Learning Optimising Shaping Algorithm (ROSA), an automated reward shaping framework in which the shaping-reward function is constructed in a Markov game between two agents. A reward-shaping agent (Shaper) uses switching controls to determine which states to add shaping rewards for more efficient learning while the other agent (Controller) learns the optimal policy for the task using these shaped rewards. We prove that ROSA, which adopts existing RL algorithms, learns to construct a shaping-reward function that is beneficial to the task thus ensuring efficient convergence to high performance policies. We demonstrate ROSA’s properties in three didactic experiments and show its superior performance against state-of-the-art RS algorithms in challenging sparse reward environments.}, number={10}, journal={Proceedings of the AAAI Conference on Artificial Intelligence}, author={Mguni, David and Jafferjee, Taher and Wang, Jianhong and Perez-Nieves, Nicolas and Song, Wenbin and Tong, Feifei and Taylor, Matthew and Yang, Tianpei and Dai, Zipeng and Chen, Hui and Zhu, Jiangcheng and Shao, Kun and Wang, Jun and Yang, Yaodong}, year={2023}, month={Jun.}, pages={11604-11612} }