Q-functionals for Value-Based Continuous Control
Samuel Lobel, Sreehari Rammohan, Bowen He, Shangqun Yu, George Konidaris
Abstract
We present Q-functionals, an alternative architecture for continuous control deep reinforcement learning. Instead of returning a single value for a state-action pair, our network transforms a state into a function that can be rapidly evaluated in parallel for many actions, allowing us to efficiently choose high-value actions through sampling. This contrasts with the typical architecture of off-policy continuous control, where a policy network is trained for the sole purpose of selecting actions from the Q-function. We represent our action-dependent Q-function as a weighted sum of basis functions (Fourier, Polynomial, etc) over the action space, where the weights are state-dependent and output by the Q-functional network. Fast sampling makes practical a variety of techniques that require Monte-Carlo integration over Q-functions, and enables action-selection strategies besides simple value-maximization. We characterize our framework, describe various implementations of Q-functionals, and demonstrate strong performance on a suite of continuous control tasks.
BibTeX
@article{Lobel_Rammohan_He_Yu_Konidaris_2023, title={Q-functionals for Value-Based Continuous Control}, volume={37}, url={https://ojs.aaai.org/index.php/AAAI/article/view/26073}, DOI={10.1609/aaai.v37i7.26073}, abstractNote={We present Q-functionals, an alternative architecture for continuous control deep reinforcement learning. Instead of returning a single value for a state-action pair, our network transforms a state into a function that can be rapidly evaluated in parallel for many actions, allowing us to efficiently choose high-value actions through sampling. This contrasts with the typical architecture of off-policy continuous control, where a policy network is trained for the sole purpose of selecting actions from the Q-function. We represent our action-dependent Q-function as a weighted sum of basis functions (Fourier, Polynomial, etc) over the action space, where the weights are state-dependent and output by the Q-functional network. Fast sampling makes practical a variety of techniques that require Monte-Carlo integration over Q-functions, and enables action-selection strategies besides simple value-maximization. We characterize our framework, describe various implementations of Q-functionals, and demonstrate strong performance on a suite of continuous control tasks.}, number={7}, journal={Proceedings of the AAAI Conference on Artificial Intelligence}, author={Lobel, Samuel and Rammohan, Sreehari and He, Bowen and Yu, Shangqun and Konidaris, George}, year={2023}, month={Jun.}, pages={8932-8939} }