Sigurdur Orn Adalgeirsson

Learning Optimal Advantage from Preferences and Mistaking It for Reward

We consider algorithms for learning reward functions from human preferences over pairs of trajectory segments, as used in reinforcement learning from human feedback (RLHF). Most recent work assumes that human preferences are generated based only upon the reward accrued within those segments, or thei…