Near-Minimax-Optimal Distributional Reinforcement Learning with a Generative Model
Mark Rowland, Li Kevin Wenliang, Remi Munos, Clare Lyle, Yunhao Tang, Will Dabney
Abstract
We propose a new algorithm for model-based distributional reinforcement learning (RL), and prove that it is minimax-optimal for approximating return distributions in the generative model regime (up to logarithmic factors), the first result of this kind for any distributional RL algorithm. Our analysis also provides new theoretical perspectives on categorical approaches to distributional RL, as well as introducing a new distributional Bellman equation, the stochastic categorical CDF Bellman equation, which we expect to be of independent interest. Finally, we provide an experimental study comparing a variety of model-based distributional RL algorithms, with several key takeaways for practitioners.
BibTeX
@inproceedings{
rowland2024nearminimaxoptimal,
title={Near-Minimax-Optimal Distributional Reinforcement Learning with a Generative Model},
author={Mark Rowland and Li Kevin Wenliang and Remi Munos and Clare Lyle and Yunhao Tang and Will Dabney},
booktitle={The Thirty-eighth Annual Conference on Neural Information Processing Systems},
year={2024},
url={https://openreview.net/forum?id=JXKbf1d4ib}
}