Antonio Álvarez López

Perceptrons and Localization of Attention’s Mean-Field Landscape

The forward pass of a Transformer can be seen as an interacting particle system on the unit sphere: time plays the role of layers, particles that of token embeddings, and the unit sphere idealizes layer normalization. In some weight settings the system can even be seen as a gradient flow for an expl…