Are We Learning Yet? A Meta Review of Evaluation Failures Across Machine Learning
Thomas Liao, Rohan Taori, Inioluwa Deborah Raji, Ludwig Schmidt
Abstract
Many subfields of machine learning share a common stumbling block: evaluation. Advances in machine learning often evaporate under closer scrutiny or turn out to be less widely applicable than originally hoped. We conduct a meta-review of 107 survey papers from natural language processing, recommender systems, computer vision, reinforcement learning, computational biology, graph learning, and more, organizing the wide range of surprisingly consistent critique into a concrete taxonomy of observed failure modes. Inspired by measurement and evaluation theory, we divide failure modes into two categories: internal and external validity. Internal validity issues pertain to evaluation on a learning problem in isolation, such as improper comparisons to baselines or overfitting from test set re-use. External validity relies on relationships between different learning problems, for instance, whether progress on a learning problem translates to progress on seemingly related tasks.
BibTeX
@inproceedings{
liao2021are,
title={Are We Learning Yet? A Meta Review of Evaluation Failures Across Machine Learning},
author={Thomas Liao and Rohan Taori and Inioluwa Deborah Raji and Ludwig Schmidt},
booktitle={Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2)},
year={2021},
url={https://openreview.net/forum?id=mPducS1MsEK}
}