2020
UniLMv2: Pseudo-Masked Language Models for Unified Language Model Pre-Training
ICML 2020poster
We propose to pre-train a unified language model for both autoencoding and partially autoregressive language modeling tasks using a novel training procedure, referred to as a pseudo-masked language model (PMLM). Given an input text with masked tokens, we rely on conventional masks to learn inter-rel…