ICML 2021spotlight18 citations
Online Policy Gradient for Model Free Learning of Linear Quadratic Regulators with $\sqrt$T Regret
Abstract
We consider the task of learning to control a linear dynamical system under fixed quadratic costs, known as the Linear Quadratic Regulator (LQR) problem. While model-free approaches are often favorable in practice, thus far only model-based methods, which rely on costly system identification, have been shown to achieve regret that scales with the optimal dependence on the time horizon T. We present the first model-free algorithm that achieves similar regret guarantees. Our method relies on an efficient policy gradient scheme, and a novel and tighter analysis of the cost of exploration in policy space in this setting.
BibTeX
@InProceedings{pmlr-v139-cassel21a,
title = {Online Policy Gradient for Model Free Learning of Linear Quadratic Regulators with $\sqrt{}$T Regret},
author = {Cassel, Asaf B and Koren, Tomer},
booktitle = {Proceedings of the 38th International Conference on Machine Learning},
pages = {1304--1313},
year = {2021},
editor = {Meila, Marina and Zhang, Tong},
volume = {139},
series = {Proceedings of Machine Learning Research},
month = {18--24 Jul},
publisher = {PMLR},
pdf = {http://proceedings.mlr.press/v139/cassel21a/cassel21a.pdf},
url = {https://proceedings.mlr.press/v139/cassel21a.html},
abstract = {We consider the task of learning to control a linear dynamical system under fixed quadratic costs, known as the Linear Quadratic Regulator (LQR) problem. While model-free approaches are often favorable in practice, thus far only model-based methods, which rely on costly system identification, have been shown to achieve regret that scales with the optimal dependence on the time horizon T. We present the first model-free algorithm that achieves similar regret guarantees. Our method relies on an efficient policy gradient scheme, and a novel and tighter analysis of the cost of exploration in policy space in this setting.}
}