Model-Free Trajectory Optimization for Reinforcement Learning
Riad Akrour, Gerhard Neumann, Hany Abdulsamad, Abbas Abdolmaleki
Abstract
Many of the recent Trajectory Optimization algorithms alternate between local approximation of the dynamics and conservative policy update. However, linearly approximating the dynamics in order to derive the new policy can bias the update and prevent convergence to the optimal policy. In this article, we propose a new model-free algorithm that backpropagates a local quadratic time-dependent Q-Function, allowing the derivation of the policy update in closed form. Our policy update ensures exact KL-constraint satisfaction without simplifying assumptions on the system dynamics demonstrating improved performance in comparison to related Trajectory Optimization algorithms linearizing the dynamics.
BibTeX
@InProceedings{pmlr-v48-akrour16,
title = {Model-Free Trajectory Optimization for Reinforcement Learning},
author = {Akrour, Riad and Neumann, Gerhard and Abdulsamad, Hany and Abdolmaleki, Abbas},
booktitle = {Proceedings of The 33rd International Conference on Machine Learning},
pages = {2961--2970},
year = {2016},
editor = {Balcan, Maria Florina and Weinberger, Kilian Q.},
volume = {48},
series = {Proceedings of Machine Learning Research},
address = {New York, New York, USA},
month = {20--22 Jun},
publisher = {PMLR},
pdf = {http://proceedings.mlr.press/v48/akrour16.pdf},
url = {https://proceedings.mlr.press/v48/akrour16.html},
abstract = {Many of the recent Trajectory Optimization algorithms alternate between local approximation of the dynamics and conservative policy update. However, linearly approximating the dynamics in order to derive the new policy can bias the update and prevent convergence to the optimal policy. In this article, we propose a new model-free algorithm that backpropagates a local quadratic time-dependent Q-Function, allowing the derivation of the policy update in closed form. Our policy update ensures exact KL-constraint satisfaction without simplifying assumptions on the system dynamics demonstrating improved performance in comparison to related Trajectory Optimization algorithms linearizing the dynamics.}
}