2022
Sparse Distillation: Speeding Up Text Classification by Using Bigger Student Models
NAACL 2022long
Distilling state-of-the-art transformer models into lightweight student models is an effective way to reduce computation cost at inference time. The student models are typically compact transformers with fewer parameters, while expensive operations such as self-attention persist. Therefore, the impr…