Telescoping Density-Ratio Estimation
Benjamin Rhodes, Kai Xu, Michael U. Gutmann
Abstract
Density-ratio estimation via classification is a cornerstone of unsupervised learning. It has provided the foundation for state-of-the-art methods in representation learning and generative modelling, with the number of use-cases continuing to proliferate. However, it suffers from a critical limitation: it fails to accurately estimate ratios p/q for which the two densities differ significantly. Empirically, we find this occurs whenever the KL divergence between p and q exceeds tens of nats. To resolve this limitation, we introduce a new framework, telescoping density-ratio estimation (TRE), that enables the estimation of ratios between highly dissimilar densities in high-dimensional spaces. Our experiments demonstrate that TRE can yield substantial improvements over existing single-ratio methods for mutual information estimation, representation learning and energy-based modelling.
BibTeX
@inproceedings{NEURIPS2020_33d3b157,
author = {Rhodes, Benjamin and Xu, Kai and Gutmann, Michael U.},
booktitle = {Advances in Neural Information Processing Systems},
editor = {H. Larochelle and M. Ranzato and R. Hadsell and M.F. Balcan and H. Lin},
pages = {4905--4916},
publisher = {Curran Associates, Inc.},
title = {Telescoping Density-Ratio Estimation},
url = {https://proceedings.neurips.cc/paper_files/paper/2020/file/33d3b157ddc0896addfb22fa2a519097-Paper.pdf},
volume = {33},
year = {2020}
}