Rickard Brännvall

The Inhibitor: ReLU and Addition-Based Attention for Efficient Transformers (Student Abstract)

To enhance the computational efficiency of quantized Transformers, we replace the dot-product and Softmax-based attention with an alternative mechanism involving addition and ReLU activation only. This side-steps the expansion to double precision often required by matrix multiplication and avoids co…