Progressively Knowledge Distillation via Re-parameterizing Diffusion Reverse Process
Knowledge distillation aims at transferring knowledge from the teacher model to the student one by aligning their distributions. Feature-level distillation often uses L2 distance or its variants as the loss function, based on the assumption that outputs follow normal distributions. This poses a si…