BraVE: Offline Reinforcement Learning for Discrete Combinatorial Action Spaces
Matthew Landers, Taylor W. Killian, Hugo Barnes, Thomas Hartvigsen, Afsaneh Doryab
Abstract
Offline reinforcement learning in high-dimensional, discrete action spaces is challenging due to the exponential scaling of the joint action space with the number of sub-actions and the complexity of modeling sub-action dependencies. Existing methods either exhaustively evaluate the action space, making them computationally infeasible, or factorize Q-values, failing to represent joint sub-action effects. We propose \textbf{Bra}nch \textbf{V}alue \textbf{E}stimation (BraVE), a value-based method that uses tree-structured action traversal to evaluate a linear number of joint actions while preserving dependency structure. BraVE outperforms prior offline RL methods by up to $20\times$ in environments with over four million actions.
BibTeX
@inproceedings{
landers2025brave,
title={Bra{VE}: Offline Reinforcement Learning for Discrete Combinatorial Action Spaces},
author={Matthew Landers and Taylor W. Killian and Hugo Barnes and Thomas Hartvigsen and Afsaneh Doryab},
booktitle={The Thirty-ninth Annual Conference on Neural Information Processing Systems},
year={2025},
url={https://openreview.net/forum?id=Oj5tVkjbHD}
}