Reinforcement learning model balances reward and cognitive effort
Policy Complexity, Reaction Time, and Bounded Rationality in Reinforcement Learning
Machine LearningArtificial IntelligenceInformation Theory
Summary
Humans cannot think and act with unlimited mental resources, so their decisions depend on how complex and costly it is to use certain information. The authors created a learning method called MI-SARSA that takes into account the mental effort needed to use specific information when making choices. This method not only predicts what choices will be made but also how long those choices might take. By balancing the benefits of using information against its costs, this approach explains why people sometimes choose simpler strategies that get results faster but may be less accurate.
What this means in practice
- •For cognitive modelers: Model human decision-making times and policy complexity by incorporating information costs into learning algorithms.
- •For human-computer interaction designers: Design adaptive systems that account for user cognitive limitations by predicting reaction times based on informational effort.
Authors
James Wu, Chris R. Sims
Abstract
Biological agents do not learn under conditions of unlimited computation. For humans, learning and choice are shaped by constraints on perception, attention, and working memory, which limit how much state information guides behavior and therefore bound policy complexity. Standard reinforcement learning models typically optimize reward without explicitly representing these internal costs, making them less suitable as models of biological intelligence. We derive MI-SARSA, an on-policy temporal-difference algorithm that incorporates mutual-information regularization through a learned marginal action prior and a penalty on state-specific deviations from that prior. This yields a sequential learning model in which state information is used selectively when its expected return benefit justifies the added informational cost. Critically, the same state-specific information cost that governs policy compression also generates trial-level predictions for reaction time, distinguishing MI-SARSA from most reinforcement learning models, which predict choices or returns but not latency. Empirically, MI-SARSA produces a reward-complexity tradeoff, and stronger information penalties produce simpler policies with lower control costs and faster reaction times. Under environment shift, increasing regularization reduces post-switch performance degradation but also lowers asymptotic return, revealing a robustness-capacity tradeoff. Together, these results position MI-SARSA as a model of bounded sequential learning under cognitive constraints.