2026
One Bias After Another: Mechanistic Reward Shaping and Persistent Biases in Language Reward Models
ICML 2026poster
Reward Models (RMs) are crucial for online alignment of language models (LMs) with human preferences. However, RM-based preference-tuning is vulnerable to reward hacking, whereby LM policies learn undesirable behaviors from flawed RMs. By systematically measuring biases in five high-quality RMs, inc…