Cyril Letrouit

2 accepted papers

2026

Token Sample Complexity of Attention

Léa Bohbot, Cyril Letrouit, Gabriel Peyré, François-Xavier Vialard

ICML 2026poster

As context windows in large language models continue to expand, it is essential to characterize how attention behaves at extreme sequence lengths. We introduce token-sample complexity: the rate at which attention computed on $n$ tokens converges to its infinite-token limit. We estimate finite-$n$ co…

Cited by 0 Source ↗Scholar ↗

2023

The emergence of clusters in self-attention dynamics

Borjan Geshkovski, Cyril Letrouit, Yury Polyanskiy, Philippe Rigollet

NeurIPS 2023poster

Viewing Transformers as interacting particle systems, we describe the geometry of learned representations when the weights are not time-dependent. We show that particles, representing tokens, tend to cluster toward particular limiting objects as time tends to infinity. Using techniques from dynamica…

Cited by 61 Source ↗PDF ↗code ↗Scholar ↗