Convergence of an actor-critic gradient flow for entropy regularised MDPs in general spaces

Denis Zorba; David Siska; Lukasz Szpruch

ICLR 2026poster0 citations

Convergence of an actor-critic gradient flow for entropy regularised MDPs in general spaces

Denis Zorba, David Siska, Lukasz Szpruch

Abstract

We prove the stability and global convergence of a coupled actor-critic gradient flow for infinite-horizon and entropy-regularised Markov decision processes (MDPs) in continuous state and action space with linear function approximation under Q-function realisability. We consider a version of the actor critic gradient flow where the critic is updated using temporal difference (TD) learning while the policy is updated using a policy mirror descent method on a separate timescale. For general action spaces, the relative entropy regularizer is unbounded and thus it is not clear a priori that the actor-critc flow does not suffer from finite-time blow-up. Therefore we first demonstrate stability which in turn enables us obtain a convergence rate of the actor critic flow to the optimal regularised value function. The arguments presented show that timescale separation is crucial for stability and convergence in this setting.

Reinforcement LearningGradient FlowMarkov Decision ProcessEntropy RegularizationNon-convex optimizationMirror descent methodFisher–Rao gradient flowGlobal convergenceFunction approximationActor Critic

BibTeX

@inproceedings{
zorba2026convergence,
title={Convergence of Actor-Critic gradient flow for entropy regularised {MDP}s in general spaces},
author={Denis Zorba and David Siska and Lukasz Szpruch},
booktitle={The Fourteenth International Conference on Learning Representations},
year={2026},
url={https://openreview.net/forum?id=KUlPxDQF3T}
}