DORM: Preference Data Weights Optimization for Reward Modeling in LLM Alignment
Rongzhi Zhang, Chenwei Zhang, Xinyang Zhang, Liang Qiu, Haoming Jiang, Yuchen Zhuang, Qingru Zhang, Hyokun Yun
Abstract
Aligning large language models (LLMs) with human preferences relies heavily on high-quality reward models. However, existing approaches struggle with two critical challenges: noisy preference labels and the varying importance of preference samples. We introduce DORM, a method that enhances reward modeling by learning to dynamically weigh preference data.DORM initializes data importance using a combination of model uncertainty and prediction disagreement, then iteratively refines them via bilevel optimization to maximize validation performance. Using only 50k samples, DORM trains a 12B reward model that achieves 90.5% accuracy on RewardBench, matching the performance of models trained on significantly larger datasets. Furthermore, downstream alignment tasks show that fine-tuned LLMs with DORM achieve a 61.2% win rate against baseline methods, highlighting its data efficiency and generalizability.
BibTeX
@inproceedings{emnlp2025_dormpreferenceda,
title = {DORM: Preference Data Weights Optimization for Reward Modeling in LLM Alignment},
author = {Rongzhi Zhang and Chenwei Zhang and Xinyang Zhang and Liang Qiu and Haoming Jiang and Yuchen Zhuang and Qingru Zhang and Hyokun Yun and Xian Li and Bing Yin and Tuo Zhao and Chao Zhang},
booktitle = {EMNLP 2025},
year = {2025}
}