NeurIPS 2020poster173 citations
On the Value of Out-of-Distribution Testing: An Example of Goodhart's Law
Damien Teney, Ehsan Abbasnejad, Kushal Kafle, Robik Shrestha, Christopher Kanan, Anton van den Hengel
Abstract
Out-of-distribution (OOD) testing is increasingly popular for evaluating a machine learning system's ability to generalize beyond the biases of a training set. OOD benchmarks are designed to present a different joint distribution of data and labels between training and test time. VQA-CP has become the standard OOD benchmark for visual question answering, but we discovered three troubling practices in its current use. First, most published methods rely on explicit knowledge of the construction of the OOD splits. They often rely on
BibTeX
@inproceedings{NEURIPS2020_045117b0,
author = {Teney, Damien and Abbasnejad, Ehsan and Kafle, Kushal and Shrestha, Robik and Kanan, Christopher and van den Hengel, Anton},
booktitle = {Advances in Neural Information Processing Systems},
editor = {H. Larochelle and M. Ranzato and R. Hadsell and M.F. Balcan and H. Lin},
pages = {407--417},
publisher = {Curran Associates, Inc.},
title = {On the Value of Out-of-Distribution Testing: An Example of Goodhart\textquotesingle s Law},
url = {https://proceedings.neurips.cc/paper_files/paper/2020/file/045117b0e0a11a242b9765e79cbf113f-Paper.pdf},
volume = {33},
year = {2020}
}