Rahul Vashisht

Faster Query-Key Learning Sharpens Attention in Self-Attention Models

A standard self-attention layer consists of two interacting circuits: the query-key circuit that governs attention allocation, and the output-value circuit that maps attended representations to predictions. Collapsed and factorized parameterizations of the query-key and output-value circuits lead to…