2023
How does the task complexity of masked pretraining objectives affect downstream performance?
ACL 2023findings
Masked language modeling (MLM) is a widely used self-supervised pretraining objective, where a model needs to predict an original token that is replaced with a mask given contexts. Although simpler and computationally efficient pretraining objectives, e.g., predicting the first character of a masked…