VA-learning as a more efficient alternative to Q-learning

Yunhao Tang; Remi Munos; Mark Rowland; Michal Valko

ICML 2023poster7 citations

VA-learning as a more efficient alternative to Q-learning

Yunhao Tang, Remi Munos, Mark Rowland, Michal Valko

Abstract

In reinforcement learning, the advantage function is critical for policy improvement, but is often extracted from a learned Q-function. A natural question is: Why not learn the advantage function directly? In this work, we introduce VA-learning, which directly learns advantage function and value function using bootstrapping, without explicit reference to Q-functions. VA-learning learns off-policy and enjoys similar theoretical guarantees as Q-learning. Thanks to the direct learning of advantage function and value function, VA-learning improves the sample efficiency over Q-learning both in tabular implementations and deep RL agents on Atari-57 games. We also identify a close connection between VA-learning and the dueling architecture, which partially explains why a simple architectural change to DQN agents tends to improve performance.

BibTeX

@inproceedings{icml2023_valearningasamor,
  title = {VA-learning as a more efficient alternative to Q-learning},
  author = {Yunhao Tang and Remi Munos and Mark Rowland and Michal Valko},
  booktitle = {ICML 2023},
  year = {2023}
}