Decoding Decoders: Finding Optimal Representation Spaces for Unsupervised Similarity Tasks
Vitalii Zhelezniak, Dan Busbridge, April Shen, Samuel L. Smith, Nils Y. Hammerla
Abstract
Experimental evidence indicates that simple models outperform complex deep networks on many unsupervised similarity tasks. Introducing the concept of an optimal representation space, we provide a simple theoretical resolution to this apparent paradox. In addition, we present a straightforward procedure that, without any retraining or architectural modifications, allows deep recurrent models to perform equally well (and sometimes better) when compared to shallow models. To validate our analysis, we conduct a set of consistent empirical evaluations and introduce several new sentence embedding models in the process. Even though this work is presented within the context of natural language processing, the insights are readily applicable to other domains that rely on distributed representations for transfer tasks.
BibTeX
@misc{
zhelezniak2018decoding,
title={Decoding Decoders: Finding Optimal Representation Spaces for Unsupervised Similarity Tasks},
author={Vitalii Zhelezniak and Dan Busbridge and April Shen and Samuel L. Smith and Nils Y. Hammerla},
year={2018},
url={https://openreview.net/forum?id=Byd-EfWCb},
}