Do You Really Need to Pretrain Q-Functions for Online RL Fine-Tuning?
2026-07-29 • Machine Learning
Machine Learning
AI summaryⓘ
The authors studied whether it helps to pretrain the Q-function in reinforcement learning when you already have a pretrained policy. They found that just pretraining the Q-function usually doesn't improve results much because it learns values linked to the old policy, not the one used during fine-tuning. To fix this, the authors introduced a method called Initialization via Policy Ensemble (IPE), which uses multiple different policies to better train the Q-function. In tests on control tasks, IPE improved fine-tuning performance compared to standard Q-function pretraining.
reinforcement learningQ-functionpretrainingfine-tuningpolicyoffline dataonline RLvalue-based RLpolicy ensemblecontinuous control
Authors
Perry Dong, Ron Polonsky, Dorsa Sadigh, Chelsea Fin
Abstract
Pre-training followed by fine-tuning has become the dominant recipe for learning performant policies, and in value-based reinforcement learning (RL) this raises a natural question: given a pretrained policy, should the Q-function be pretrained on offline data too? Conventional wisdom suggests it should, but recent results show that online RL with a randomly-initialized Q-function can result in highly performant and reliable policies without needing to pretrain the Q-function. In this paper, we systematically study whether pretraining the Q-function actually helps when fine-tuning on top of a pretrained base policy. We find, surprisingly, that naive Q-function pretraining often provides little benefit over random initialization. We show this stems from a fundamental mismatch: the Q-function learned during pretraining targets the pretrained policy's Q-function, not the Q-function that online fine-tuning converges to, and this gap persists even after offline value maximization. Motivated by this finding, we propose Initialization via Policy Ensemble (IPE), a simple method that trains multiple diverse policies and uses their pooled rollouts to bootstrap the Q-function learning in online RL. Across a suite of challenging continuous control benchmarks, IPE yields an average 1.26x improvement in fine-tuning performance over naive Q-function pre-training.