2023
Recipes for Sequential Pre-training of Multilingual Encoder and Seq2Seq Models
ACL 2023findings
Pre-trained encoder-only and sequence-to-sequence (seq2seq) models each have advantages, however training both model types from scratch is computationally expensive. We explore recipes to improve pre-training efficiency by initializing one model from the other. (1) Extracting the encoder from a seq2…