Entropy regularization improves investment and spending decisions under market changes

Entropy-Regularized Reinforcement Learning for the Regime-Switching Consumption-Investment Problem

Machine Learning

Summary

Planning how to spend and invest money during changing market conditions is hard. This paper introduces a new way to make better, more flexible choices by adding a kind of randomness (called entropy regularization) to the decision process. The researchers found exact formulas for how to mix and match investment and spending strategies when markets switch between different states. Their approach also smoothly matches traditional investment methods as the randomness goes away. They tested their method in simulated markets and saw it learns steadily and reliably.

What this means in practice

  • For financial technology developers: Develop algorithms for dynamic portfolio and consumption strategies that adapt to changing market regimes using entropy-regularized reinforcement learning.
  • For quantitative investment teams: Use theoretical distribution forms to stabilize and speed up learning of consumption and allocation strategies in regime-switching market models.

Authors

Congde Hu, Danping Li, Xun Li, Yijie Zeng

Abstract

This paper establishes an entropy-regularized reinforcement learning (ERRL) framework for the continuous-time consumption-investment problem under a regime-switching market model with general utility functions. By incorporating randomized exploratory policies and Shannon entropy regularization into the stochastic optimal control formulation, we derive closed-form representations for the optimal exploratory consumption and investment strategies. Specifically, the optimal investment strategy follows a multivariate Gaussian distribution, whereas the optimal consumption strategy takes the form of a Gibbs measure---which specializes to a Gamma distribution under logarithmic utility and a non-standard exponential distribution under power utility. Crucially, our theoretical analysis reveals that the classical deterministic optimal decisions correspond to the mode---rather than the mean---of these exploratory distributions. Furthermore, we establish the convergence of the exploratory value function to its classical unregularized counterpart as the temperature parameter vanishes. For algorithmic design, we develop a continuous-time q-learning framework grounded in a novel continuous-time q-function, prove the policy improvement theorem, and establish martingale characterizations for both on-policy and off-policy learning. We find that parameterizing the actor network based on theoretical distribution families can help accelerate convergence and enhance numerical stability. Furthermore, numerical experiments across regime-switching environments provide empirical support for our theoretical findings, demonstrating stable loss convergence as well as the monotonicity and concavity of the learned value functions.