2025
Attention Sinks: A 'Catch, Tag, Release' Mechanism for Embeddings
NeurIPS 2025poster
Large language models (LLMs) often concentrate their attention on a few specific tokens referred to as *attention sinks*. Common examples include the first token, a prompt-independent sink, and punctuation tokens, which are prompt-dependent. While the tokens causing the sinks often lack direct seman…