Shirou Jing

InfoDLM: an Information-Adaptive Framework for Discrete Diffusion Language Model Pretraining

Diffusion language models (DLMs) can match or surpass similarly sized autoregressive language models on language understanding and reasoning. However, their mask-and-denoise pretraining relies on heuristic random masking, which fails to target the most informative tokens. Consequently, the model spe…