Blanka Kövér

A Framework for Understanding Learnability in Transformers

Transformers consistently fail to learn certain simple functions such as Parity---which returns whether the input has an even number of ones---even when they can provably compute them with specific parameter settings. This gap between *learnability* and *expressivity* is particularly prominent for s…