Representation Matters: Assessing the Importance of Subgroup Allocations in Training Data
Esther Rolf, Theodora T Worledge, Benjamin Recht, Michael Jordan
Abstract
Collecting more diverse and representative training data is often touted as a remedy for the disparate performance of machine learning predictors across subpopulations. However, a precise framework for understanding how dataset properties like diversity affect learning outcomes is largely lacking. By casting data collection as part of the learning process, we demonstrate that diverse representation in training data is key not only to increasing subgroup performances, but also to achieving population-level objectives. Our analysis and experiments describe how dataset compositions influence performance and provide constructive results for using trends in existing data, alongside domain knowledge, to help guide intentional, objective-aware dataset design
BibTeX
@InProceedings{pmlr-v139-rolf21a,
title = {Representation Matters: Assessing the Importance of Subgroup Allocations in Training Data},
author = {Rolf, Esther and Worledge, Theodora T and Recht, Benjamin and Jordan, Michael},
booktitle = {Proceedings of the 38th International Conference on Machine Learning},
pages = {9040--9051},
year = {2021},
editor = {Meila, Marina and Zhang, Tong},
volume = {139},
series = {Proceedings of Machine Learning Research},
month = {18--24 Jul},
publisher = {PMLR},
pdf = {http://proceedings.mlr.press/v139/rolf21a/rolf21a.pdf},
url = {https://proceedings.mlr.press/v139/rolf21a.html},
abstract = {Collecting more diverse and representative training data is often touted as a remedy for the disparate performance of machine learning predictors across subpopulations. However, a precise framework for understanding how dataset properties like diversity affect learning outcomes is largely lacking. By casting data collection as part of the learning process, we demonstrate that diverse representation in training data is key not only to increasing subgroup performances, but also to achieving population-level objectives. Our analysis and experiments describe how dataset compositions influence performance and provide constructive results for using trends in existing data, alongside domain knowledge, to help guide intentional, objective-aware dataset design}
}