Agentic reinforcement learning improved by managing multi session trajectories
Mosaic: Multi-Session Agentic Reinforcement Learning
Computation and LanguageArtificial IntelligenceMachine Learning
Summary
Most current reinforcement learning methods expect all interactions to happen in one continuous session, but real-world tasks often involve multiple sessions. The authors find that treating all sessions the same ignores how each contributes differently to overall progress. They propose a new method called Mosaic that estimates the value of each session more fairly and optimizes learning accordingly. Tests show Mosaic performs better than existing approaches, using fewer sessions and making better decisions about how to split tasks across sessions.
What this means in practice
- •For software engineers: Improve reinforcement learning agents that operate over multiple interrupted sessions, increasing efficiency and accuracy in task completion.
- •For dialog system developers: Enhance conversational AI by better managing user interactions that span multiple sessions, improving response quality and resource use.
Authors
Yilong Xu, Minrui Xu, Xinyu Zhou, Xiang Long, Haochen Tan, Zhijiang Guo
Abstract
Most existing agentic reinforcement learning methods assume single-session trajectories, where interaction proceeds within one continuously accumulated context. However, agentic techniques such as context compaction and subagent forking may produce trajectories spanning multiple sessions, creating a mismatch between the single-session training assumption and inference-time execution. Therefore, we introduce Multi-Session Agentic Reinforcement Learning, a more general setting that supports an arbitrary number of sessions within a trajectory. Under this setting, a widely adopted method is to broadcast trajectory-level advantages to all sessions, but this does not distinguish contributions from various sessions or guide session allocation within trajectories. To address this issue, we propose Mosaic, a multi-session agentic RL method with critic-free hierarchical optimization. At the trajectory level, it incorporates session-usage signals into advantage estimation. At the session level, it shapes each session's advantage according to the estimated marginal contribution. Experiments on BrowseComp with Qwen3.5-4B show that Mosaic outperforms broadcasting in both context compaction and subagent forking scenarios while using fewer sessions on average. Further analyses show that our method achieves improvements in session utility and allocation, which we suggested is a promising target for future advances in multi-session agentic RL.