Aditya Anantharaman

ReAugKD: Retrieval-Augmented Knowledge Distillation For Pre-trained Language Models

Knowledge Distillation (KD) is one of the most effective approaches to deploying large-scale pre-trained language models in low-latency environments by transferring the knowledge contained in the large-scale models to smaller student models. Prior KD approaches use the soft labels and intermediate a…