AISTATS 2020poster6 citations
A Three Sample Hypothesis Test for Evaluating Generative Models
Casey Meehan, Kamalika Chaudhuri, Sanjoy Dasgupta
Abstract
Detecting overfitting in generative models is an important challenge in machine learning. In this work, we formalize a form of overfitting that we call {\em{data-copying}} – where the generative model memorizes and outputs training samples or small variations thereof. We provide a three sample test for detecting data-copying that uses the training set, a separate sample from the target distribution, and a generated sample from the model, and study the performance of our test on several canonical models and datasets.
BibTeX
@InProceedings{pmlr-v108-meehan20a,
title = {A Three Sample Hypothesis Test for Evaluating Generative Models},
author = {Meehan, Casey and Chaudhuri, Kamalika and Dasgupta, Sanjoy},
booktitle = {Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics},
pages = {3546--3556},
year = {2020},
editor = {Chiappa, Silvia and Calandra, Roberto},
volume = {108},
series = {Proceedings of Machine Learning Research},
month = {26--28 Aug},
publisher = {PMLR},
pdf = {http://proceedings.mlr.press/v108/meehan20a/meehan20a.pdf},
url = {https://proceedings.mlr.press/v108/meehan20a.html},
abstract = {Detecting overfitting in generative models is an important challenge in machine learning. In this work, we formalize a form of overfitting that we call {\em{data-copying}} – where the generative model memorizes and outputs training samples or small variations thereof. We provide a three sample test for detecting data-copying that uses the training set, a separate sample from the target distribution, and a generated sample from the model, and study the performance of our test on several canonical models and datasets.}
}