Zhiling Zhou

How Transformers Represent Hierarchies: A Local-to-Global Mechanism

Large language models built on autoregressive Transformers excel at next-token prediction, but it is unclear how their internal computations capture the latent hierarchical dependencies that often underlie language. We study this question in a controlled formal-language setting based on probabilisti…