Mengrou Shou

Difference-Masking: Choosing What to Mask in Continued Pretraining

The self-supervised objective of masked prediction has led to promising performance gains on a variety of downstream tasks. However, while most approaches randomly mask tokens, there is strong intuition that deciding what to mask can substantially improve learning outcomes. We investigate this in co…