2024
On the Role of Attention Masks and LayerNorm in Transformers
NeurIPS 2024poster
Self-attention is the key mechanism of transformers, which are the essential building blocks of modern foundation models. Recent studies have shown that pure self-attention suffers from an increasing degree of rank collapse as depth increases, limiting model expressivity and further utilization of m…