Richard Sutcliffe

Enhancing Task-Specific Distillation in Small Data Regimes through Language Generation

Large-scale pretrained language models have led to significant improvements in Natural Language Processing. Unfortunately, they come at the cost of high computational and storage requirements that complicate their deployment on low-resource devices. This issue can be addressed by distilling knowledg…