Trimformer: A Novel Sequence Compression Mechanism with Local Attention

Abstract

Training on extremely long sequences poses significant challenges for attention mechanisms. In this paper, we introduce a novel trim attention mechanism that capitalizes on the inherent sparsity within attention processes. This mechanism effectively compresses the sequence length, thereby reducing the overall computational complexity without altering the number of trainable parameters. Our experimental results demonstrate that this approach not only decreases computational demands but also outperforms the Vision Transformer in image classification tasks. The trim attention mechanism can seamlessly replace any standard attention layer.