Do Transformer Modifications Transfer Across Implementations and Applications?
Sharan Narang, Hyung Won Chung, Yi Tay, Liam Fedus, Thibault Fevry, Michael Matena, Karishma Malkan, Noah Fiedel
Abstract
The research community has proposed copious modifications to the Transformer architecture since it was introduced over three years ago, relatively few of which have seen widespread adoption. In this paper, we comprehensively evaluate many of these modifications in a shared experimental setting that covers most of the common uses of the Transformer in natural language processing. Surprisingly, we find that most modifications do not meaningfully improve performance. Furthermore, most of the Transformer variants we found beneficial were either developed in the same codebase that we used or are relatively minor changes. We conjecture that performance improvements may strongly depend on implementation details and correspondingly make some recommendations for improving the generality of experimental results.
BibTeX
@inproceedings{narang-etal-2021-transformer,
title = "Do Transformer Modifications Transfer Across Implementations and Applications?",
author = "Narang, Sharan and
Chung, Hyung Won and
Tay, Yi and
Fedus, Liam and
Fevry, Thibault and
Matena, Michael and
Malkan, Karishma and
Fiedel, Noah and
Shazeer, Noam and
Lan, Zhenzhong and
Zhou, Yanqi and
Li, Wei and
Ding, Nan and
Marcus, Jake and
Roberts, Adam and
Raffel, Colin",
editor = "Moens, Marie-Francine and
Huang, Xuanjing and
Specia, Lucia and
Yih, Scott Wen-tau",
booktitle = "Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing",
month = nov,
year = "2021",
address = "Online and Punta Cana, Dominican Republic",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2021.emnlp-main.465/",
doi = "10.18653/v1/2021.emnlp-main.465",
pages = "5758--5773"
}