Alessio Giorlandino

Two failure modes of deep transformers and how to avoid them: a unified theory of signal propagation at initialisation

Finding the right initialisation for neural networks is crucial to ensure smooth training and good performance. In transformers, the wrong initialisation can lead to one of two failure modes of self-attention layers: rank collapse, where all tokens collapse into similar representations, and entropy…