Quantile Credit Assignment
Thomas Mesnard, Wenqi Chen, Alaa Saade, Yunhao Tang, Mark Rowland, Theophane Weber, Clare Lyle, Audrunas Gruslys
Abstract
In reinforcement learning, the credit assignment problem is to distinguish luck from skill, that is, separate the inherent randomness in the environment from the controllable effects of the agent's actions. This paper proposes two novel algorithms, Quantile Credit Assignment (QCA) and Hindsight QCA (HQCA), which incorporate distributional value estimation to perform credit assignment. QCA uses a network that predicts the quantiles of the return distribution, whereas HQCA additionally incorporates information about the future. Both QCA and HQCA have the appealing interpretation of leveraging an estimate of the quantile level of the return (interpreted as the level of "luck") in order to derive a "luck-dependent" baseline for policy gradient methods. We show theoretically that this approach gives an unbiased policy gradient estimate that can yield significant variance reductions over a standard value estimate baseline. QCA and HQCA significantly outperform prior state-of-the-art methods on a range of extremely difficult credit assignment problems.
BibTeX
@inproceedings{icml2023_quantilecreditas,
title = {Quantile Credit Assignment},
author = {Thomas Mesnard and Wenqi Chen and Alaa Saade and Yunhao Tang and Mark Rowland and Theophane Weber and Clare Lyle and Audrunas Gruslys and Michal Valko and Will Dabney and Georg Ostrovski and Eric Moulines and Remi Munos},
booktitle = {ICML 2023},
year = {2023}
}