Beyond the Bellman Recursion: A Pontryagin-Guided Framework for Non-Exponential Discounting

Abstract

Most value-based and actor-critic reinforcement learning methods rely on Bellman-style recursions, yet these recursions collapse under non-exponential discounting common in human preferences and survival processes. We show the breakdown is structural: exponential discounting sits at a fragile intersection of multiplicativity and time homogeneity, and violating either property breaks standard dynamic programming. To overcome this, we propose **Pontryagin-Guided Direct Policy Optimization (PG-DPO)**, a variational framework that abandons recursion and couples the Pontryagin Maximum Principle with Monte Carlo rollouts via an *Adjoint-MC projection* enforcing pointwise Hamiltonian maximization. Across multidimensional hyperbolic and survival-discount benchmarks, PG-DPO improves accuracy and stability where equation-driven solvers and critic-based baselines diverge.