Risk-Sensitive Policy Optimization via Predictive CVaR Policy Gradient

Abstract

This paper addresses a policy optimization task with the conditional value-at-risk (CVaR) objective. We introduce the *predictive CVaR policy gradient*, a novel approach that seamlessly integrates risk-neutral policy gradient algorithms with minimal modifications. Our method incorporates a reweighting strategy in gradient calculation -- individual cost terms are reweighted in proportion to their *predicted* contribution to the objective. These weights can be easily estimated through a separate learning procedure. We provide theoretical and empirical analyses, demonstrating the validity and effectiveness of our proposed method.