Simplifying Reward Design in Complex Robotics: Average-Reward Maximum Entropy Reinforcement Learning

Abstract

This paper presents a novel approach to addressing the control challenges of underactuated systems, focusing on the swing-up and stabilisation tasks on the double pendulum system. We propose the Average-Reward Entropy Advantage Policy Optimisation (AR-EAPO), a model-free reinforcement learning (RL) algorithm that integrates the strengths of the average-reward RL and the maximum entropy RL (MaxEnt RL). The average reward criterion allows the use of a simple reward function by naturally promoting the longterm goals, at the same time MaxEnt RL encourages the robustness of the policy. We validate our approach through simulations, consistently outperforming standard RL baselines and traditional control methods. Also, we provide preliminary test results on real double pendulum hardware. Additional experiments on MuJoCo environments further demonstrate AR-EAPO's efficacy on general continuous control tasks. This work underscores the potential of the average-reward criterion in simplifying control design while achieving superior results.