Interaction aware correction improves quantized multi turn reinforcement learning
InterQ-RL: Interaction-Aware Correction of Execution Mismatch in Quantized Agentic RL
Machine Learning
Summary
Multi-step agent tasks in reinforcement learning often require lots of computing power, which can be reduced by using lower precision calculations called quantization. However, this quantization can cause differences between what the learning model expects and what the agent actually does, especially over many steps. The authors introduce a method called InterQ-RL that detects and corrects these differences by looking at the entire sequence of actions, not just the current step. This approach improves the performance of quantized agents to nearly match high-precision ones without needing extra heavy computation.
What this means in practice
- •For ai system developers: Reduce computing costs and maintain high accuracy when deploying multi-turn reinforcement learning agents with quantized models.
- •For cloud service engineers: Optimize resource use and speed for cloud-based intelligent agents interacting continuously with users or environments using quantized RL with execution correction.
Authors
Yingbo Hao, Song Liu, Guanghua Yu, Zhaojing Zhou, Jianchen Zhu, Yi Zou
Abstract
Multi-turn agent reinforcement learning (RL) relies on long closed-loop rollouts, making generation and KV-cache costs a major bottleneck. Quantized rollout reduces these costs, but its low-precision executor also serves as the behavior policy and can induce execution mismatch with the learner. Existing quantized-RL methods are designed primarily for single-turn generation. They align the learner forward with quantized rollout or correct the current-token rollout-learner ratio, without modeling the policy history that produces each agent state. Even after quantization alignment, residual rollout-learner mismatch accumulates along the state-reaching policy history, so a current action may look locally aligned despite a mismatched history. We introduce InterQ-RL (Interaction-aware Quantized Reinforcement Learning) to correct this post-alignment mismatch. It applies bounded execution-prefix correction over the state-reaching policy history, allocates correction strength across action segments by effective support under a fixed displacement budget, and requires no additional model forward pass. We evaluate dense and mixture-of-experts (MoE) backbones from 4B to 30B on ALFWorld, WebShop, and ScienceWorld, with all quantized arms sharing the same rollout with 8-bit weights, activations, and KV cache (W8A8-KV8). Across four environment-backbone settings, InterQ-RL achieves the strongest quantized endpoint result, matching BF16 to within 0.7 points and exceeding it in two settings. It matches or exceeds the learning speed of alignment alone while avoiding its late performance regression.