Fine-Tuning Discrete Diffusion Models with Policy Gradient Methods

Oussama Zekri; Nicolas Boulle

NeurIPS 2025poster0 citations

Fine-Tuning Discrete Diffusion Models with Policy Gradient Methods

Abstract

Discrete diffusion models have recently gained significant attention due to their ability to process complex discrete structures for language modeling. However, fine-tuning these models with policy gradient methods, as is commonly done in Reinforcement Learning from Human Feedback (RLHF), remains a challenging task. We propose an efficient, broadly applicable, and theoretically justified policy gradient algorithm, called Score Entropy Policy Optimization (SEPO), for fine-tuning discrete diffusion models over non-differentiable rewards. Our numerical experiments across several discrete generative tasks demonstrate the scalability and efficiency of our method.

Discrete Diffusion ModelsPolicy gradient algorithmsNon-differentiable rewardsFine-TuningReinforcement Learning from Human Feedback

BibTeX

@inproceedings{
zekri2025finetuning,
title={Fine-Tuning Discrete Diffusion Models with Policy Gradient Methods},
author={Oussama Zekri and Nicolas Boulle},
booktitle={The Thirty-ninth Annual Conference on Neural Information Processing Systems},
year={2025},
url={https://openreview.net/forum?id=rXFzVRZsbt}
}