CREPE: Controlling diffusion with REPlica Exchange
Jiajun He, Paul Jeha, Peter Potaptchik, Leo Zhang, José Miguel Hernández-Lobato, Yuanqi Du, Saifuddin Syed, Francisco Vargas
Abstract
Inference-time control of diffusion models aims to steer model outputs to satisfy new constraints without retraining. Previous approaches have mostly relied on heuristic guidance or have been coupled with Sequential Monte Carlo (SMC) for bias correction. In this paper, we propose a flexible alternative based on replica exchange, an algorithm designed initially for sampling problems. We refer to this method as the CREPE (Controlling with REPlica Exchange). Unlike SMC, CREPE: (i) generates particles sequentially, (ii) maintains high diversity in the generated samples after a burn-in period, and (iii) enables online refinement or early termination. We demonstrate its versatility across various tasks, including temperature annealing, reward tilting, model composition and classifier-free guidance debiasing, with competitive performance compared to prior SMC methods.
BibTeX
@inproceedings{
he2026crepe,
title={{CREPE}: Controlling diffusion with {REP}lica Exchange},
author={Jiajun He and Paul Jeha and Peter Potaptchik and Leo Zhang and Jos{\'e} Miguel Hern{\'a}ndez-Lobato and Yuanqi Du and Saifuddin Syed and Francisco Vargas},
booktitle={The Fourteenth International Conference on Learning Representations},
year={2026},
url={https://openreview.net/forum?id=uOZRWcbiZl}
}