ResQ: A Residual Q Function-based Approach for Multi-Agent Reinforcement Learning Value Factorization
Siqi SHEN, Mengwei Qiu, Jun Liu, Weiquan Liu, Yongquan Fu, Xinwang Liu, Cheng Wang
Abstract
The factorization of state-action value functions for Multi-Agent Reinforcement Learning (MARL) is important. Existing studies are limited by their representation capability, sample efficiency, and approximation error. To address these challenges, we propose, ResQ, a MARL value function factorization method, which can find the optimal joint policy for any state-action value function through residual functions. ResQ masks some state-action value pairs from a joint state-action value function, which is transformed as the sum of a main function and a residual function. ResQ can be used with mean-value and stochastic-value RL. We theoretically show that ResQ can satisfy both the individual global max (IGM) and the distributional IGM principle without representation limitations. Through experiments on matrix games, the predator-prey, and StarCraft benchmarks, we show that ResQ can obtain better results than multiple expected/stochastic value factorization methods.
BibTeX
@inproceedings{
shen2022resq,
title={ResQ: A Residual Q Function-based Approach for Multi-Agent Reinforcement Learning Value Factorization},
author={Siqi SHEN and Mengwei Qiu and Jun Liu and Weiquan Liu and Yongquan Fu and Xinwang Liu and Cheng Wang},
booktitle={Thirty-Sixth Conference on Neural Information Processing Systems},
year={2022},
url={https://openreview.net/forum?id=bdnZ_1qHLCW}
}