2021
H-Transformer-1D: Fast One-Dimensional Hierarchical Attention for Sequences
ACL 2021long
We describe an efficient hierarchical method to compute attention in the Transformer architecture. The proposed attention mechanism exploits a matrix structure similar to the Hierarchical Matrix (H-Matrix) developed by the numerical analysis community, and has linear run time and memory complexity.…