ICLR 2024poster257 citations
Universal Guidance for Diffusion Models
Arpit Bansal, Hong-Min Chu, Avi Schwarzschild, Roni Sengupta, Micah Goldblum, Jonas Geiping, Tom Goldstein
Abstract
Typical diffusion models are trained to accept a particular form of conditioning, most commonly text, and cannot be conditioned on other modalities without retraining. In this work, we propose a universal guidance algorithm that enables diffusion models to be controlled by arbitrary guidance modalities without the need to retrain any use-specific components. We show that our algorithm successfully generates quality images with guidance functions including segmentation, face recognition, object detection, style guidance and classifier signals.
Generative ModelsComputer VisionDiffusion Models
BibTeX
@inproceedings{
bansal2024universal,
title={Universal Guidance for Diffusion Models},
author={Arpit Bansal and Hong-Min Chu and Avi Schwarzschild and Roni Sengupta and Micah Goldblum and Jonas Geiping and Tom Goldstein},
booktitle={The Twelfth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=pzpWBbnwiJ}
}