Attention Mechanism with Energy-Friendly Operations
Yu Wan, Baosong Yang, Dayiheng Liu, Rong Xiao, Derek Wong, Haibo Zhang, Boxing Chen, Lidia Chao
Abstract
Attention mechanism has become the dominant module in natural language processing models. It is computationally intensive and depends on massive power-hungry multiplications. In this paper, we rethink variants of attention mechanism from the energy consumption aspects. After reaching the conclusion that the energy costs of several energy-friendly operations are far less than their multiplication counterparts, we build a novel attention model by replacing multiplications with either selective operations or additions. Empirical results on three machine translation tasks demonstrate that the proposed model, against the vanilla one, achieves competitable accuracy while saving 99% and 66% energy during alignment calculation and the whole attention procedure. Our code will be released upon the acceptance.
BibTeX
@inproceedings{wan-etal-2022-attention,
title = "Attention Mechanism with Energy-Friendly Operations",
author = "Wan, Yu and
Yang, Baosong and
Liu, Dayiheng and
Xiao, Rong and
Wong, Derek and
Zhang, Haibo and
Chen, Boxing and
Chao, Lidia",
editor = "Muresan, Smaranda and
Nakov, Preslav and
Villavicencio, Aline",
booktitle = "Findings of the Association for Computational Linguistics: ACL 2022",
month = may,
year = "2022",
address = "Dublin, Ireland",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2022.findings-acl.313/",
doi = "10.18653/v1/2022.findings-acl.313",
pages = "3969--3976"
}