AISTATS 2021poster40 citations
Mirrorless Mirror Descent: A Natural Derivation of Mirror Descent
Suriya Gunasekar, Blake Woodworth, Nathan Srebro
Abstract
We present a direct (primal only) derivation of Mirror Descent as a “partial” discretization of gradient flow on a Riemannian manifold where the metric tensor is the Hessian of the Mirror Descent potential function. We contrast this discretization to Natural Gradient Descent, which is obtained by a “full” forward Euler discretization. This view helps shed light on the relationship between the methods and allows generalizing Mirror Descent to any Riemannian geometry in $\mathbb{R}^d$, even when the metric tensor is not a Hessian, and thus there is no “dual.”
BibTeX
@InProceedings{pmlr-v130-gunasekar21a,
title = { Mirrorless Mirror Descent: A Natural Derivation of Mirror Descent },
author = {Gunasekar, Suriya and Woodworth, Blake and Srebro, Nathan},
booktitle = {Proceedings of The 24th International Conference on Artificial Intelligence and Statistics},
pages = {2305--2313},
year = {2021},
editor = {Banerjee, Arindam and Fukumizu, Kenji},
volume = {130},
series = {Proceedings of Machine Learning Research},
month = {13--15 Apr},
publisher = {PMLR},
pdf = {http://proceedings.mlr.press/v130/gunasekar21a/gunasekar21a.pdf},
url = {https://proceedings.mlr.press/v130/gunasekar21a.html},
abstract = { We present a direct (primal only) derivation of Mirror Descent as a “partial” discretization of gradient flow on a Riemannian manifold where the metric tensor is the Hessian of the Mirror Descent potential function. We contrast this discretization to Natural Gradient Descent, which is obtained by a “full” forward Euler discretization. This view helps shed light on the relationship between the methods and allows generalizing Mirror Descent to any Riemannian geometry in $\mathbb{R}^d$, even when the metric tensor is not a Hessian, and thus there is no “dual.” }
}