Marginalized Operators for Off-policy Reinforcement Learning
Yunhao Tang, Mark Rowland, Remi Munos, Michal Valko
Abstract
In this work, we propose marginalized operators, a new class of off-policy evaluation operators for reinforcement learning. Marginalized operators strictly generalize generic multi-step operators, such as Retrace, as special cases. Marginalized operators also suggest a form of sample-based estimates with potential variance reduction, compared to sample-based estimates of the original multi-step operators. We show that the estimates for marginalized operators can be computed in a scalable way, which also generalizes prior results on marginalized importance sampling as special cases. Finally, we empirically demonstrate that marginalized operators provide performance gains to off-policy evaluation problems and downstream policy optimization algorithms.
BibTeX
@InProceedings{pmlr-v151-tang22a,
title = { Marginalized Operators for Off-policy Reinforcement Learning },
author = {Tang, Yunhao and Rowland, Mark and Munos, Remi and Valko, Michal},
booktitle = {Proceedings of The 25th International Conference on Artificial Intelligence and Statistics},
pages = {655--679},
year = {2022},
editor = {Camps-Valls, Gustau and Ruiz, Francisco J. R. and Valera, Isabel},
volume = {151},
series = {Proceedings of Machine Learning Research},
month = {28--30 Mar},
publisher = {PMLR},
pdf = {https://proceedings.mlr.press/v151/tang22a/tang22a.pdf},
url = {https://proceedings.mlr.press/v151/tang22a.html},
abstract = { In this work, we propose marginalized operators, a new class of off-policy evaluation operators for reinforcement learning. Marginalized operators strictly generalize generic multi-step operators, such as Retrace, as special cases. Marginalized operators also suggest a form of sample-based estimates with potential variance reduction, compared to sample-based estimates of the original multi-step operators. We show that the estimates for marginalized operators can be computed in a scalable way, which also generalizes prior results on marginalized importance sampling as special cases. Finally, we empirically demonstrate that marginalized operators provide performance gains to off-policy evaluation problems and downstream policy optimization algorithms. }
}