Timothy Norman

Modelling Attention with Aitchison Geometry: Token Distinguishability and Temperature Scaling

The attention mechanism with softmax normalisation is a foundational component of Transformer-based large language models. However, with very long contexts, attention scores are known to diminish, raising fundamental questions about token distinguishability and how it can be preserved. In this work,…