Provably Efficient Neural GTD for Off-Policy Learning

Hoi-To Wai; Zhuoran Yang; Zhaoran Wang; Mingyi Hong

NeurIPS 2020poster9 citations

Provably Efficient Neural GTD for Off-Policy Learning

Hoi-To Wai, Zhuoran Yang, Zhaoran Wang, Mingyi Hong

Abstract

This paper studies a gradient temporal difference (GTD) algorithm using neural network (NN) function approximators to minimize the mean squared Bellman error (MSBE). For off-policy learning, we show that the minimum MSBE problem can be recast into a min-max optimization involving a pair of over-parameterized primal-dual NNs. The resultant formulation can then be tackled using a neural GTD algorithm. We analyze the convergence of the proposed algorithm with a 2-layer ReLU NN architecture using $m$ neurons and prove that it computes an approximate optimal solution to the minimum MSBE problem as $m \rightarrow \infty$.

BibTeX

@inproceedings{NEURIPS2020_75ebb02f,
 author = {Wai, Hoi-To and Yang, Zhuoran and Wang, Zhaoran and Hong, Mingyi},
 booktitle = {Advances in Neural Information Processing Systems},
 editor = {H. Larochelle and M. Ranzato and R. Hadsell and M.F. Balcan and H. Lin},
 pages = {10431--10442},
 publisher = {Curran Associates, Inc.},
 title = {Provably Efficient Neural GTD for Off-Policy Learning},
 url = {https://proceedings.neurips.cc/paper_files/paper/2020/file/75ebb02f92fc30a8040bbd625af999f1-Paper.pdf},
 volume = {33},
 year = {2020}
}