GeoDE: a Geographically Diverse Evaluation Dataset for Object Recognition
Vikram V. Ramaswamy, Sing Yu Lin, Dora Zhao, Aaron Bryan Adcock, Laurens van der Maaten, Deepti Ghadiyaram, Olga Russakovsky
Abstract
Current dataset collection methods typically scrape large amounts of data from the web. While this technique is extremely scalable, data collected in this way tends to reinforce stereotypical biases, can contain personally identifiable information, and typically originates from Europe and North America. In this work, we rethink the dataset collection paradigm and introduce GeoDE, a geographically diverse dataset with 61,940 images from 40 classes and 6 world regions, and no personally identifiable information, collected by soliciting images from people across the world. We analyse GeoDE to understand differences in images collected in this manner compared to web-scraping. Despite the smaller size of this dataset, we demonstrate its use as both an evaluation and training dataset, allowing us to highlight shortcomings in current models, as well as demonstrate improved performance even when training on this small dataset. We release the full dataset and code at https://geodiverse-data-collection.cs.princeton.edu/
BibTeX
@inproceedings{
ramaswamy2023geode,
title={Geo{DE}: a Geographically Diverse Evaluation Dataset for Object Recognition},
author={Vikram V. Ramaswamy and Sing Yu Lin and Dora Zhao and Aaron Bryan Adcock and Laurens van der Maaten and Deepti Ghadiyaram and Olga Russakovsky},
booktitle={Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track},
year={2023},
url={https://openreview.net/forum?id=JGVSxwKHbq}
}