Integrating Dependency Tree into Self-Attention for Sentence Representation
Junhua Ma, Jiajun Li, Yuxuan Liu, Shangbo Zhou, Xue Li
Abstract
Recent progress on parse tree encoder for sentence representation learning is notable. However, these works mainly en-code tree structures recursively, which is not conducive to parallelization. On the other hand, these works rarely take into account the labels of arcs in dependency trees. To address both issues, we propose Dependency-Transformer, which applies a relation-attention mechanism that works in concert with the self-attention mechanism. This mechanism aims to encode the dependency and the spatial positional relations between nodes in the dependency tree of sentences. By a score-based method, we successfully inject the syntax information without affecting Transformer’s parallelizability. Our model outperforms or is comparable to the state-of-the-art methods on four tasks for sentence representation and has obvious advantages in computational efficiency.
BibTeX
@inproceedings{icassp2022_integratingdepen,
title = {Integrating Dependency Tree into Self-Attention for Sentence Representation},
author = {Junhua Ma and Jiajun Li and Yuxuan Liu and Shangbo Zhou and Xue Li},
booktitle = {ICASSP 2022},
year = {2022}
}