ParetoHqD: Fast Offline Multiobjective Alignment of Large Language Models Using Pareto High-Quality Data

Haoran Gu; Handing Wang; Yi Mei; Mengjie Zhang; Yaochu Jin

AAAI 2026technical0 citations

ParetoHqD: Fast Offline Multiobjective Alignment of Large Language Models Using Pareto High-Quality Data

Haoran Gu, Handing Wang, Yi Mei, Mengjie Zhang, Yaochu Jin

Abstract

Aligning large language models with multiple human expectations and values is crucial for ensuring that they adequately serve a variety of user needs. To this end, offline multiobjective alignment algorithms such as the Rewards-in-Context algorithm have shown strong performance and efficiency. However, inappropriate preference representations and training with imbalanced reward scores limit the performance of such algorithms. In this work, we introduce ParetoHqD that addresses the above issues by representing human preferences as preference directions in the objective space and regarding data near the Pareto front as

BibTeX

@inproceedings{aaai2026_paretohqdfastoff,
  title = {ParetoHqD: Fast Offline Multiobjective Alignment of Large Language Models Using Pareto High-Quality Data},
  author = {Haoran Gu and Handing Wang and Yi Mei and Mengjie Zhang and Yaochu Jin},
  booktitle = {AAAI 2026},
  year = {2026}
}