Three Steps at a Time: Learning Representations from Action Sequences in Contrastive RL
2026-08-31 • Machine Learning
Machine Learning
AI summaryⓘ
The authors studied how reinforcement learning can improve by looking at chunks of actions rather than one action at a time. They extended a method called contrastive reinforcement learning (CRL) to use these action chunks and saw big improvements in performance on various tests. While usually benefits come from understanding longer sequences or better reward predictions, the authors found that for CRL, the main reason is that action chunks give clearer hints about the goal. This helps the system learn better representations and makes it work much better overall.
reinforcement learningself-supervised learningcontrastive reinforcement learningaction chunksrepresentation learningoffline learningonline learningMarkov propertycritic networkmulti-step returns
Authors
Michal Korniak, Kamil Dybek, Benjamin Eysenbach, Marco Bagatella, Michał Bortkiewicz
Abstract
While self-supervised approaches to reinforcement learning have achieved strong results by learning representations of states and actions, a key open question is the time scale over which actions should be modeled. Departing from the standard formulation relying on single-step actions, we extend contrastive reinforcement learning (CRL), a prototypical self-supervised method, to operate over action chunks, and find that this results in large, pervasive gains across established offline and online benchmarks: +31.7% and +93.1% across 18 and 11 environments respectively. While action-chunking-driven gains are generally explained through the ability to model non-Markovian, temporally extended policies, and to propagate unbiased multi-step returns, interestingly, we find that these arguments only partially apply to CRL. Our empirical studies suggest that, in the context of CRL, an action chunk carries more information about the goal than a single action, measurably improving the critic's representations, and rendering the algorithm significantly more effective.