CGDD: Contrastive Gaussian-Dirac Diffusion Model

Abstract

This paper proposes the contrastive Gaussian-Dirac diffusion (CGDD) model, which introduces a novel approach to hybrid noise diffusion by combining continuous and discrete noise processes for text generation tasks. Leveraging the contrastive learning, the proposed CGDD structures the embedding space based on the word frequency, promoting an easy-first generation approach that aligns with the observed frequency of word usage. Furthermore, a new approximation is presented for the loss function in the continuous-discrete diffusion setting, addressing the limitations of the previous models that rely solely on the Gaussian distributions. Experimental results demonstrate that the proposed approaches are effective based on the evaluations over multiple measurements.