Ran Dou

Trimformer: A Novel Sequence Compression Mechanism with Local Attention

Training on extremely long sequences poses significant challenges for attention mechanisms. In this paper, we introduce a novel trim attention mechanism that capitalizes on the inherent sparsity within attention processes. This mechanism effectively compresses the sequence length, thereby reducing t…