Instant flow Q-learning speeds up offline reinforcement learning
Compositional Average Velocity Modeling for Efficient Diffusion Offline Reinforcement Learning
Machine Learning
Summary
Offline reinforcement learning lets computers learn from past experiences without new exploration but can be slow and costly to train and run. The paper introduces Instant Flow Q-Learning (InsQL), a new method that creates actions in a single step rather than many, making training and use faster. The authors achieve this by changing how the method models the decision-making process to work consistently across different levels and combine it with a smart optimization approach. Their tests show it is as good or better than existing methods while being much quicker in both learning and action generation.
What this means in practice
- •For robotic system engineers: Generate robot control actions faster and more efficiently during offline training for manipulation tasks without iterative sampling overhead.
- •For machine learning engineers: Deploy offline reinforcement learning policies with significantly reduced computational cost at inference time by using single-step action generation.
Authors
Thanh Nguyen, Abu Hanif Muhammad Syarubany, Hwanhee Kim, Chang D. Yoo
Abstract
Diffusion-based policies have recently emerged as a powerful paradigm for offline reinforcement learning, but their reliance on multi-step denoising introduces substantial computational overhead and requires backpropagation through time during training, limiting scalability and practical deployment. In this work, we propose \textbf{Instant Flow Q-Learning (InsQL)}, a single-step generative policy learning framework that retains the expressive capacity of diffusion models while eliminating iterative sampling and BPTT. InsQL builds on the flow-matching perspective of diffusion policies, but replaces marginal velocity parameterization with \emph{average-velocity} modeling and introduces a principled compositional mechanism, yielding a theoretically grounded objective that learns average velocity consistently across scales. This formulation provides a direct noise-to-action mapping and enables single-step action generation. InsQL further unifies compositional average-velocity modeling with behavior-regularized actor--critic optimization in a single framework, requiring only a single policy network and avoiding distillation, multi-stage procedures, and Jacobian computations during training. At inference time, actions are generated in a single forward pass and can be refined via value-guided candidate selection to improve performance. Extensive experiments on standard offline RL benchmarks and real-world robotic manipulation tasks show that InsQL matches or surpasses state-of-the-art diffusion-based methods while substantially reducing both training and inference time. Overall, these results establish compositional average-velocity modeling as an efficient and scalable alternative to multi-step generative policies for offline RL.