Q-target learning improves reinforcement policies with weak data
From Weak Data to Strong Policy: Q-Targets Enable Provable In-Context Reinforcement Learning
Machine Learning
Summary
When teaching computers to learn from experience, poor or random past examples can confuse them. The authors propose a new method that helps the computer use rewards more effectively instead of just copying past actions. This approach makes the learning more reliable, especially when the past data isn't very good. They show this works well both in theory and on some computer simulation tests.
What this means in practice
- •For robotics engineers: Train robots to improve control policies using suboptimal past demonstrations by focusing on reward signals instead of imitation alone.
- •For game ai developers: Enhance reinforcement learning agents to better utilize weak or random gameplay data by estimating action values rather than copying actions.
Authors
Yichen Lin, Xuyuan Xiong, Xue Wang, Xiangfu Meng, Mike Mingcheng Wei, Tao Yao
Abstract
Existing in-context reinforcement learning methods mainly pretrain Transformers with supervised behavior-prediction objectives. This enables task inference from context, but makes the learned policy strongly depend on the quality of offline actions: when trajectories are weak or suboptimal, imitation itself becomes a biased learning signal. We propose Q-Target Pretrained Transformers (QTPT), which keeps the context-conditioned Transformer architecture but replaces behavior cloning with a Bellman-style Q-target objective. QTPT therefore learns to use rewards and transitions in the context to estimate action values, rather than simply imitating the behavior policy. We theoretically analyze QTPT in stochastic linear bandits and finite-horizon MDPs, showing stronger robustness to data quality than supervised pretraining. Empirically, QTPT improves over supervised behavior prediction on controlled RL benchmarks with random or suboptimal data, and we examine extensions to D4RL Kitchen and AntMaze. Supplementary experiments evaluate backbone robustness, meta-RL comparisons, task-coherent context, and unsupported-action value overestimation. These comparisons distinguish the benefits of Q-target pretraining from the remaining limitations of offline coverage.