Distilling Diffusion Models into Conditional GANs
MinGuk Kang*, Richard Zhang, Connelly Barnes, Sylvain Paris, Suha Kwak, Jaesik Park, Eli Shechtman, Jun-Yan Zhu
Abstract
"We propose a method to distill a complex multistep diffusion model into a single-step conditional GAN student model, dramatically accelerating inference, while preserving image quality. Our approach interprets diffusion distillation as a paired image-to-image translation task, using noise-to-image pairs of the diffusion model’s ODE trajectory. For efficient regression loss computation, we propose E-LatentLPIPS, a perceptual loss operating directly in diffusion model’s latent space, utilizing an ensemble of augmentations. Furthermore, we adapt a diffusion model to construct a multi-scale discriminator with a text alignment loss to build an effective conditional GAN-based formulation. E-LatentLPIPS converges more efficiently than many existing distillation methods, even accounting for dataset construction costs. We demonstrate that our one-step generator outperforms cutting-edge one-step diffusion distillation models – SDXL-Turbo and SDXL-Lightning – on the COCO benchmark."
BibTeX
@inproceedings{eccv2024_distillingdiffus,
title = {Distilling Diffusion Models into Conditional GANs},
author = {MinGuk Kang* and Richard Zhang and Connelly Barnes and Sylvain Paris and Suha Kwak and Jaesik Park and Eli Shechtman and Jun-Yan Zhu and Taesung Park*},
booktitle = {ECCV 2024},
year = {2024}
}