Brian Summa

Attention Sinks in Diffusion Transformers: A Causal Analysis

Attention sinks (tokens that receive disproportionate attention mass) are often assumed to be functionally important in autoregressive language models. Whether such sinks are necessary in diffusion transformers remains unclear. We present a causal analysis of attention sinks in text-to-image diffusi…