Critic Regularized Regression
Ziyu Wang, Alexander Novikov, Konrad Zolna, Josh S Merel, Jost Tobias Springenberg, Scott E Reed, Bobak Shahriari, Noah Siegel
Abstract
Offline reinforcement learning (RL), also known as batch RL, offers the prospect of policy optimization from large pre-recorded datasets without online environment interaction. It addresses challenges with regard to the cost of data collection and safety, both of which are particularly pertinent to real-world applications of RL. Unfortunately, most off-policy algorithms perform poorly when learning from a fixed dataset. In this paper, we propose a novel offline RL algorithm to learn policies from data using a form of critic-regularized regression (CRR). We find that CRR performs surprisingly well and scales to tasks with high-dimensional state and action spaces -- outperforming several state-of-the-art offline RL algorithms by a significant margin on a wide range of benchmark tasks.
BibTeX
@inproceedings{NEURIPS2020_588cb956,
author = {Wang, Ziyu and Novikov, Alexander and Zolna, Konrad and Merel, Josh S and Springenberg, Jost Tobias and Reed, Scott E and Shahriari, Bobak and Siegel, Noah and Gulcehre, Caglar and Heess, Nicolas and de Freitas, Nando},
booktitle = {Advances in Neural Information Processing Systems},
editor = {H. Larochelle and M. Ranzato and R. Hadsell and M.F. Balcan and H. Lin},
pages = {7768--7778},
publisher = {Curran Associates, Inc.},
title = {Critic Regularized Regression},
url = {https://proceedings.neurips.cc/paper_files/paper/2020/file/588cb956d6bbe67078f29f8de420a13d-Paper.pdf},
volume = {33},
year = {2020}
}