Jake Cunningham

Contribution Weights: A Geometrical Analysis of Self-Attention Transformers

Analyzing attention weights has become a standard approach for interpreting the information flow of Large Language Models (LLMs). However, this approach has significant limitations as it neglects the geometric properties of the value vectors being aggregated. To address this gap, we introduce \emph{…