Estimating Generalization under Distribution Shifts via Domain-Invariant Representations
Ching-Yao Chuang, Antonio Torralba, Stefanie Jegelka
Abstract
When machine learning models are deployed on a test distribution different from the training distribution, they can perform poorly, but overestimate their performance. In this work, we aim to better estimate a model’s performance under distribution shift, without supervision. To do so, we use a set of domain-invariant predictors as a proxy for the unknown, true target labels. Since the error of the resulting risk estimate depends on the target risk of the proxy model, we study generalization of domain-invariant representations and show that the complexity of the latent representation has a significant influence on the target risk. Empirically, our approach (1) enables self-tuning of domain adaptation models, and (2) accurately estimates the target error of given models under distribution shift. Other applications include model selection, deciding early stopping and error detection.
BibTeX
@InProceedings{pmlr-v119-chuang20a,
title = {Estimating Generalization under Distribution Shifts via Domain-Invariant Representations},
author = {Chuang, Ching-Yao and Torralba, Antonio and Jegelka, Stefanie},
booktitle = {Proceedings of the 37th International Conference on Machine Learning},
pages = {1984--1994},
year = {2020},
editor = {III, Hal Daumé and Singh, Aarti},
volume = {119},
series = {Proceedings of Machine Learning Research},
month = {13--18 Jul},
publisher = {PMLR},
pdf = {http://proceedings.mlr.press/v119/chuang20a/chuang20a.pdf},
url = {https://proceedings.mlr.press/v119/chuang20a.html},
abstract = {When machine learning models are deployed on a test distribution different from the training distribution, they can perform poorly, but overestimate their performance. In this work, we aim to better estimate a model’s performance under distribution shift, without supervision. To do so, we use a set of domain-invariant predictors as a proxy for the unknown, true target labels. Since the error of the resulting risk estimate depends on the target risk of the proxy model, we study generalization of domain-invariant representations and show that the complexity of the latent representation has a significant influence on the target risk. Empirically, our approach (1) enables self-tuning of domain adaptation models, and (2) accurately estimates the target error of given models under distribution shift. Other applications include model selection, deciding early stopping and error detection.}
}