NeurIPS 2016poster191 citations
On Multiplicative Integration with Recurrent Neural Networks
Yuhuai Wu, Saizheng Zhang, Ying Zhang, Yoshua Bengio, Ruslan Salakhutdinov
Abstract
We introduce a general simple structural design called “Multiplicative Integration” (MI) to improve recurrent neural networks (RNNs). MI changes the way of how the information flow gets integrated in the computational building block of an RNN, while introducing almost no extra parameters. The new structure can be easily embedded into many popular RNN models, including LSTMs and GRUs. We empirically analyze its learning behaviour and conduct evaluations on several tasks using different RNN models. Our experimental results demonstrate that Multiplicative Integration can provide a substantial performance boost over many of the existing RNN models.
BibTeX
@inproceedings{NIPS2016_f69e505b,
author = {Wu, Yuhuai and Zhang, Saizheng and Zhang, Ying and Bengio, Yoshua and Salakhutdinov, Russ R},
booktitle = {Advances in Neural Information Processing Systems},
editor = {D. Lee and M. Sugiyama and U. Luxburg and I. Guyon and R. Garnett},
pages = {},
publisher = {Curran Associates, Inc.},
title = {On Multiplicative Integration with Recurrent Neural Networks},
url = {https://proceedings.neurips.cc/paper_files/paper/2016/file/f69e505b08403ad2298b9f262659929a-Paper.pdf},
volume = {29},
year = {2016}
}