Z. Wang

GOLD: Generalized Knowledge Distillation via Out-of-Distribution-Guided Language Data Generation

Knowledge distillation from LLMs is essential for the efficient deployment of language models. Prior works have proposed data generation using LLMs for preparing distilled models. We argue that generating data with LLMs is prone to sampling mainly from the center of original content distribution. Th…