Self-Supervised Reinforcement Learning for Out-of-Distribution Recovery via Auxiliary Reward

Abstract

Recently, the real-world applications of reinforcement learning (RL) have seen the problem of taking actions in an out-of-distribution (OOD) state. However, most existing research is limited to take actions to narrow the visited training distribution and OOD, and does not consider the efficiency to choose such actions. In this paper, we propose a novel approach, called Self-Supervised Reinforcement Learning for OOD recovery via Auxiliary Reward (SRL-AR), to address this issue. By leveraging cumulative reward, we force the representations to discriminate state-action pairs with different returns as auxiliary task. Then, the auxiliary reward calculated from the auxiliary loss is used to generate a new policy that can effectively handle OOD situations. Moreover, we show that our method outperforms prior works in terms of asymptotic performance and sample efficiency on MuJoCo tasks.