ADDQ: Adaptive distributional double Q-learning
Leif Döring, Benedikt Wille, Maximilian Birr, Mihail Bîrsan, Martin Slowik
Abstract
Bias problems in the estimation of Q-values are a well-known obstacle that slows down convergence of Q-learning and actor-critic methods. One of the reasons of the success of modern RL algorithms is partially a direct or indirect overestimation reduction mechanism. We introduce an easy to implement method built on top of distributional reinforcement learning (DRL) algorithms to deal with the overestimation in a locally adaptive way. Our framework ADDQ is simple to implement, existing DRL implementations can be improved with a few lines of code. We provide theoretical backup and experimental results in tabular, Atari, and MuJoCo environments, comparisons with state-of-the-art methods, and a proof of convergence in the tabular case.
BibTeX
@inproceedings{
doring2025addq,
title={{ADDQ}: Adaptive distributional double Q-learning},
author={Leif D{\"o}ring and Benedikt Wille and Maximilian Birr and Mihail B{\^\i}rsan and Martin Slowik},
booktitle={Forty-second International Conference on Machine Learning},
year={2025},
url={https://openreview.net/forum?id=Bpyh6H9Xr1}
}