Active Surrogate Estimators: An Active Learning Approach to Label-Efficient Model Evaluation
Jannik Kossen, Sebastian Farquhar, Yarin Gal, Tom Rainforth
Abstract
We propose Active Surrogate Estimators (ASEs), a new method for label-efficient model evaluation. Evaluating model performance is a challenging and important problem when labels are expensive. ASEs address this active testing problem using a surrogate-based estimation approach that interpolates the errors of points with unknown labels, rather than forming a Monte Carlo estimator. ASEs actively learn the underlying surrogate, and we propose a novel acquisition strategy, XWED, that tailors this learning to the final estimation task. We find that ASEs offer greater label-efficiency than the current state-of-the-art when applied to challenging model evaluation problems for deep neural networks.
BibTeX
@inproceedings{
kossen2022active,
title={Active Surrogate Estimators: An Active Learning Approach to Label-Efficient Model Evaluation},
author={Jannik Kossen and Sebastian Farquhar and Yarin Gal and Tom Rainforth},
booktitle={Advances in Neural Information Processing Systems},
editor={Alice H. Oh and Alekh Agarwal and Danielle Belgrave and Kyunghyun Cho},
year={2022},
url={https://openreview.net/forum?id=uIXyp4Ip9fG}
}