Daichi Hayakawa

Theoretical Analysis of Hierarchical Language Recognition and Generation by Transformers without Positional Encoding

In this study, we provide constructive proof that Transformers can recognize and generate hierarchical language efficiently with respect to model size, even without the need for a specific positional encoding.Specifically, we show that causal masking and a starting token enable Transformers to compu…