Convex Representation Learning for Generalized Invariance in Semi-Inner-Product Space
Yingyi Ma, Vignesh Ganapathiraman, Yaoliang Yu, Xinhua Zhang
Abstract
Invariance (defined in a general sense) has been one of the most effective priors for representation learning. Direct factorization of parametric models is feasible only for a small range of invariances, while regularization approaches, despite improved generality, lead to nonconvex optimization. In this work, we develop a \emph{convex} representation learning algorithm for a variety of generalized invariances that can be modeled as semi-norms. Novel Euclidean embeddings are introduced for kernel representers in a semi-inner-product space, and approximation bounds are established. This allows invariant representations to be learned efficiently and effectively as confirmed in our experiments, along with accurate predictions.
BibTeX
@InProceedings{pmlr-v119-ma20b,
title = {Convex Representation Learning for Generalized Invariance in Semi-Inner-Product Space},
author = {Ma, Yingyi and Ganapathiraman, Vignesh and Yu, Yaoliang and Zhang, Xinhua},
booktitle = {Proceedings of the 37th International Conference on Machine Learning},
pages = {6532--6542},
year = {2020},
editor = {III, Hal Daumé and Singh, Aarti},
volume = {119},
series = {Proceedings of Machine Learning Research},
month = {13--18 Jul},
publisher = {PMLR},
pdf = {http://proceedings.mlr.press/v119/ma20b/ma20b.pdf},
url = {https://proceedings.mlr.press/v119/ma20b.html},
abstract = {Invariance (defined in a general sense) has been one of the most effective priors for representation learning. Direct factorization of parametric models is feasible only for a small range of invariances, while regularization approaches, despite improved generality, lead to nonconvex optimization. In this work, we develop a \emph{convex} representation learning algorithm for a variety of generalized invariances that can be modeled as semi-norms. Novel Euclidean embeddings are introduced for kernel representers in a semi-inner-product space, and approximation bounds are established. This allows invariant representations to be learned efficiently and effectively as confirmed in our experiments, along with accurate predictions.}
}