2025
Born a Transformer -- Always a Transformer? On the Effect of Pretraining on Architectural Abilities
NeurIPS 2025poster
Transformers have theoretical limitations in modeling certain sequence-to-sequence tasks, yet it remains largely unclear if these limitations play a role in large-scale pretrained LLMs, or whether LLMs might effectively overcome these constraints in practice due to the scale of both the models thems…