Exploration via Epistemic Value Estimation
Simon Schmitt, John Shawe-Taylor, Hado van Hasselt
Abstract
How to efficiently explore in reinforcement learning is an open problem. Many exploration algorithms employ the epistemic uncertainty of their own value predictions -- for instance to compute an exploration bonus or upper confidence bound. Unfortunately the required uncertainty is difficult to estimate in general with function approximation. We propose epistemic value estimation (EVE): a recipe that is compatible with sequential decision making and with neural network function approximators. It equips agents with a tractable posterior over all their parameters from which epistemic value uncertainty can be computed efficiently. We use the recipe to derive an epistemic Q-Learning agent and observe competitive performance on a series of benchmarks. Experiments confirm that the EVE recipe facilitates efficient exploration in hard exploration tasks.
BibTeX
@article{Schmitt_Shawe-Taylor_van Hasselt_2023, title={Exploration via Epistemic Value Estimation}, volume={37}, url={https://ojs.aaai.org/index.php/AAAI/article/view/26164}, DOI={10.1609/aaai.v37i8.26164}, abstractNote={How to efficiently explore in reinforcement learning is an open problem. Many exploration algorithms employ the epistemic uncertainty of their own value predictions -- for instance to compute an exploration bonus or upper confidence bound. Unfortunately the required uncertainty is difficult to estimate in general with function approximation. We propose epistemic value estimation (EVE): a recipe that is compatible with sequential decision making and with neural network function approximators. It equips agents with a tractable posterior over all their parameters from which epistemic value uncertainty can be computed efficiently. We use the recipe to derive an epistemic Q-Learning agent and observe competitive performance on a series of benchmarks. Experiments confirm that the EVE recipe facilitates efficient exploration in hard exploration tasks.}, number={8}, journal={Proceedings of the AAAI Conference on Artificial Intelligence}, author={Schmitt, Simon and Shawe-Taylor, John and van Hasselt, Hado}, year={2023}, month={Jun.}, pages={9742-9751} }