LPPG-RL: Lexicographically Projected Policy Gradient Reinforcement Learning with Subproblem Exploration
Ruiyu Qiu, Rui Wang, Guanghui Yang, Xiang Li, Zhijiang Shao
Abstract
Lexicographic multi-objective problems, which consist of multiple conflicting subtasks with explicit priorities, are common in real-world applications. Despite the advantages of Reinforcement Learning (RL) in single tasks, extending conventional RL methods to prioritized multiple objectives remains challenging. In particular, traditional Safe RL and Multi-Objective RL (MORL) methods have difficulty enforcing priority orderings efficiently. Therefore, Lexicographic Multi-Objective RL (LMORL) methods have been developed to address these challenges. However, existing LMORL methods either rely on heuristic threshold tuning with prior knowledge or are restricted to discrete domains. To overcome these limitations, we propose Lexicographically Projected Policy Gradient RL (LPPG-RL), a novel LMORL framework which leverages sequential gradient projections to identify feasible policy update directions, thereby enabling LPPG-RL broadly compatible with all policy gradient algorithms in continuous spaces. LPPG-RL reformulates the projection step as an optimization problem, and utilizes Dykstra
BibTeX
@inproceedings{aaai2026_lppgrllexicograp,
title = {LPPG-RL: Lexicographically Projected Policy Gradient Reinforcement Learning with Subproblem Exploration},
author = {Ruiyu Qiu and Rui Wang and Guanghui Yang and Xiang Li and Zhijiang Shao},
booktitle = {AAAI 2026},
year = {2026}
}