Fast Approximate Dynamic Programming for Infinite-Horizon Markov Decision Processes
Mohamad Amin Sharifi Kolarijani, Gyula Felix Max, Peyman Mohajerin Esfahani
Abstract
In this study, we consider the infinite-horizon, discounted cost, optimal control of stochastic nonlinear systems with separable cost and constraints in the state and input variables. Using the linear-time Legendre transform, we propose a novel numerical scheme for implementation of the corresponding value iteration (VI) algorithm in the conjugate domain. Detailed analyses of the convergence, time complexity, and error of the proposed algorithm are provided. In particular, with a discretization of size $X$ and $U$ for the state and input spaces, respectively, the proposed approach reduces the time complexity of each iteration in the VI algorithm from $O(XU)$ to $O(X+U)$, by replacing the minimization operation in the primal domain with a simple addition in the conjugate domain.
BibTeX
@inproceedings{
kolarijani2021fast,
title={Fast Approximate Dynamic Programming for Infinite-Horizon Markov Decision Processes},
author={Mohamad Amin Sharifi Kolarijani and Gyula Felix Max and Peyman Mohajerin Esfahani},
booktitle={Advances in Neural Information Processing Systems},
editor={A. Beygelzimer and Y. Dauphin and P. Liang and J. Wortman Vaughan},
year={2021},
url={https://openreview.net/forum?id=ZW-ZsleMIWk}
}