2026
Bringing Stability to Diffusion: Decomposing and Reducing Variance of Training Masked Diffusion Models
ICLR 2026poster
Masked diffusion models (MDMs) are a promising alternative to autoregressive models (ARMs), but they suffer from **inherently** much higher training variance. High variance leads to noisier gradient estimates and unstable optimization, so even equally strong pretrained MDMs and ARMs that are competi…