Adversarial Self-Attention for Language Understanding
Hongqiu Wu, Ruixue Ding, Hai Zhao, Pengjun Xie, Fei Huang, Min Zhang
Abstract
Deep neural models (e.g. Transformer) naturally learn spurious features, which create a ``shortcut'' between the labels and inputs, thus impairing the generalization and robustness. This paper advances self-attention mechanism to its robust variant for Transformer-based pre-trained language models (e.g. BERT). We propose Adversarial Self-Attention mechanism (ASA), which adversarially biases the attentions to effectively suppress the model reliance on features (e.g. specific keywords) and encourage its exploration of broader semantics. We conduct comprehensive evaluation across a wide range of tasks for both pre-training and fine-tuning stages. For pre-training, ASA unfolds remarkable performance gain compared to naive training for longer steps. For fine-tuning, ASA-empowered models outweigh naive models by a large margin considering both generalization and robustness.
BibTeX
@article{Wu_Ding_Zhao_Xie_Huang_Zhang_2023, title={Adversarial Self-Attention for Language Understanding}, volume={37}, url={https://ojs.aaai.org/index.php/AAAI/article/view/26608}, DOI={10.1609/aaai.v37i11.26608}, abstractNote={Deep neural models (e.g. Transformer) naturally learn spurious features, which create a ``shortcut’’ between the labels and inputs, thus impairing the generalization and robustness. This paper advances self-attention mechanism to its robust variant for Transformer-based pre-trained language models (e.g. BERT). We propose Adversarial Self-Attention mechanism (ASA), which adversarially biases the attentions to effectively suppress the model reliance on features (e.g. specific keywords) and encourage its exploration of broader semantics. We conduct comprehensive evaluation across a wide range of tasks for both pre-training and fine-tuning stages. For pre-training, ASA unfolds remarkable performance gain compared to naive training for longer steps. For fine-tuning, ASA-empowered models outweigh naive models by a large margin considering both generalization and robustness.}, number={11}, journal={Proceedings of the AAAI Conference on Artificial Intelligence}, author={Wu, Hongqiu and Ding, Ruixue and Zhao, Hai and Xie, Pengjun and Huang, Fei and Zhang, Min}, year={2023}, month={Jun.}, pages={13727-13735} }