Tomer M. Schlank

Looking Beyond the Top-1: Transformers Determine Top Tokens in Order

Uncovering the inner mechanisms of Transformer models offers insights into how they process and represent information. In this work, we analyze the computation performed by Transformers in the layers after the top-1 prediction remains fixed, known as the “saturation event”. We expand this concept to…