2023
Are Intermediate Layers and Labels Really Necessary? A General Language Model Distillation Method
ACL 2023findings
The large scale of pre-trained language models poses a challenge for their deployment on various devices, with a growing emphasis on methods to compress these models, particularly knowledge distillation. However, current knowledge distillation methods rely on the model’s intermediate layer features…