Improving and Accelerating Offline RL in Large Discrete Action Spaces with Structured Policy Initialization
Matthew Landers, Taylor W. Killian, Thomas Hartvigsen, Afsaneh Doryab
Abstract
Reinforcement learning in combinatorial action spaces requires searching over exponentially many joint actions to simultaneously select multiple sub-actions that form coherent combinations. Existing approaches either simplify policy learning by assuming independence across sub-actions, which often yields incoherent or invalid actions when coordination is required, or attempt to learn action structure and control jointly, which is slow and unstable. We introduce Structured Policy Initialization (SPIN), a two-stage framework that first pre-trains an Action Structure Model (ASM) to capture the manifold of valid actions, then freezes this representation and trains lightweight policy heads for control. On challenging DM Control benchmarks, SPIN improves average return by up to $39\%$ over the state of the art while reducing time to convergence by up to $12.8\times$.
BibTeX
@inproceedings{
landers2026improving,
title={Improving and Accelerating Offline {RL} in Large Discrete Action Spaces with Structured Policy Initialization},
author={Matthew Landers and Taylor W. Killian and Thomas Hartvigsen and Afsaneh Doryab},
booktitle={The Fourteenth International Conference on Learning Representations},
year={2026},
url={https://openreview.net/forum?id=rPrNaDJrLx}
}