Improved Techniques for Offline Reinforcement Learning: Advantage Value Estimation and Layernorm

Abstract

Offline reinforcement learning, which aims to learn an optimal policy from a previously collected static datasets. Due to the overestimation caused by extrapolation error, offline algorithms adopt overly pessimistic approaches, which compromise the generalization ability of the learned policy. To address these issues, we propose a method that adopts a mild constraint learning approach, comprising two components: advantage value estimation and layernorm. The first component estimates the value of actions and then selects valuable actions for imitation, which moderately relaxes the strict conservative learning and the second applies layernorm in the value function network, effectively addressing the overestimation of out-of-distribution(OOD) actions and stabilizing the training process. Experimental results on various tasks in the D4RL MuJoCo benchmark show that, compared to baseline methods, our algorithm achieves better performance in most tasks. Especially, our algorithm exhibits well generalization ability on random, medium-replay, and full-replay datasets.