Preference Optimization via Contrastive Divergence: Your Policy Is Secretly an NLL Estimator
Existing studies on preference optimization (PO) have been focused on constructing pairwise preference data following simple heuristics, such as maximizing the margin between chosen and rejected responses based on human (or AI) ratings. In this work, we develop a novel PO framework that provides th