DreamDistribution: Learning Prompt Distribution for Diverse In-distribution Generation
Brian Nlong Zhao, Yuhang Xiao, Jiashu Xu, XINYANG JIANG, Yifan Yang, Dongsheng Li, Laurent Itti, Vibhav Vineet
Abstract
The popularization of Text-to-Image (T2I) diffusion models enables the generation of high-quality images from text descriptions. However, generating diverse customized images with reference visual attributes remains challenging. This work focuses on personalizing T2I diffusion models at a more abstract concept or category level, adapting commonalities from a set of reference images while creating new instances with sufficient variations. We introduce a solution that allows a pretrained T2I diffusion model to learn a set of soft prompts, enabling the generation of novel images by sampling prompts from the learned distribution. These prompts offer text-guided editing capabilities and additional flexibility in controlling variation and mixing between multiple distributions. We also show the adaptability of the learned prompt distribution to other tasks, such as text-to-3D. Finally we demonstrate effectiveness of our approach through quantitative analysis including automatic evaluation and human assessment.
BibTeX
@inproceedings{
zhao2025dreamdistribution,
title={DreamDistribution: Learning Prompt Distribution for Diverse In-distribution Generation},
author={Brian Nlong Zhao and Yuhang Xiao and Jiashu Xu and XINYANG JIANG and Yifan Yang and Dongsheng Li and Laurent Itti and Vibhav Vineet and Yunhao Ge},
booktitle={The Thirteenth International Conference on Learning Representations},
year={2025},
url={https://openreview.net/forum?id=oQoQ4u6MQC}
}