NeurIPS 2018poster16 citations
Breaking the Activation Function Bottleneck through Adaptive Parameterization
Sebastian Flennerhag, Hujun Yin, John Keane, Mark Elliot
Abstract
Standard neural network architectures are non-linear only by virtue of a simple element-wise activation function, making them both brittle and excessively large. In this paper, we consider methods for making the feed-forward layer more flexible while preserving its basic structure. We develop simple drop-in replacements that learn to adapt their parameterization conditional on the input, thereby increasing statistical efficiency significantly. We present an adaptive LSTM that advances the state of the art for the Penn Treebank and Wikitext-2 word-modeling tasks while using fewer parameters and converging in half as many iterations.
BibTeX
@inproceedings{NEURIPS2018_cac8e130,
author = {Flennerhag, Sebastian and Yin, Hujun and Keane, John and Elliot, Mark},
booktitle = {Advances in Neural Information Processing Systems},
editor = {S. Bengio and H. Wallach and H. Larochelle and K. Grauman and N. Cesa-Bianchi and R. Garnett},
pages = {},
publisher = {Curran Associates, Inc.},
title = {Breaking the Activation Function Bottleneck through Adaptive Parameterization},
url = {https://proceedings.neurips.cc/paper_files/paper/2018/file/cac8e13055d2e4f62b6322254203b293-Paper.pdf},
volume = {31},
year = {2018}
}