2025
Dynamical Properties of Tokens in Self-Attention and Effects of Positional Encoding
NeurIPS 2025poster
This paper investigates the dynamical properties of tokens in pre-trained transformer models and explores their application to improving Transformers. To this end, we analyze the dynamical system governing the continuous-time limit of the pre-trained model and characterize the asymptotic behavior of…