João Pedro Silvestre

Consensus Is All You Get: The Role of Attention in Transformers

A key component of transformers is the attention mechanism orchestrating how each token influences the propagation of every other token along the layers of a transformer. In this paper we provide a rigorous, mathematical analysis of the asymptotic properties of attention in transformers. Although w…