Tiberiu Mușat

Mechanism and Emergence of Stacked Attention Heads in Multi-Layer Transformers

In this paper, I introduce the retrieval problem, a simple yet common reasoning task that can be solved only by transformers with a minimum number of layers, which grows logarithmically with the input size. I empirically show that large language models can solve the task under different prompting fo…