R2-SAC: A Relaxation-and-Refinement SAC Agent for Stock Portfolio Trading

Abstract

Deep reinforcement learning (DRL) has gained significant attention in stock portfolio management, in particular with the actor-critic framework that can generate continuous actions. This lends itself suitable for an end-to-end approach that produces a portfolio directly. However, there is no report that standard actor-critic agents can produce reliable excess profits in cross-market and cross-period tests. This paper introduces R<sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup>-SAC, a novel framework for stock portfolio trading that employs a ‘Relaxation and Refinement’ strategy to boost the famous Soft Actor-Critic (SAC) agent. Relaxation is employed in the stage of SAC policy generation, where the actor only produces coarse policy for individual stocks, and the real portfolio policy is formed by executing the coarse policy by a decision logic. Refinement is employed when executing the coarse policy in a real trading environment, where the decision logic takes into account turning points of the market and returns of individual stocks. Our experiments on four mainstream datasets demonstrated that R<sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup>-SAC significantly surpasses the baseline SAC in all the considered metrics, and harnesses significant excess profits in all the considered markets and test periods. Ablation studies showed that both relaxation and refinement are indispensable; they play significant roles by themselves and complement each other. The source code is available at https://github.com/AI4QTrading/R2-SAC/.