Claw-R1: A Step-Level Data Middleware System for Agentic Reinforcement Learning
2026-06-08 • Machine Learning
Machine LearningComputation and Language
AI summaryⓘ
The authors present Claw-R1, a system that helps manage the data generated when language models interact with their environment during reinforcement learning. Instead of treating interaction logs as temporary, Claw-R1 organizes them into detailed records that include all important information like prompts, responses, and rewards. It allows users to inspect and curate this data before using it to train the models, aiming to improve how data is handled in agentic reinforcement learning setups. Their work highlights the importance of managing interaction data carefully rather than just focusing on algorithms.
agentic reinforcement learninglarge language modelsdata middlewarepolicy optimizationtraining data managementinteractive agentsLLM APIreward signalsdata curationtraining pipelines
Authors
Daoyu Wang, Mingyue Cheng, Qingchuan Li, Shuo Yu, Jie Ouyang, Qi Liu
Abstract
Agentic reinforcement learning (RL) has become an important post-training paradigm for turning LLMs from static chatbots into interactive agents, giving rise to representative applications such as OpenClaw. Existing work mainly focuses on policy optimization algorithms and training frameworks, but pays less attention to the full data lifecycle of agent-environment interactions, from data production to training consumption. To bridge this gap, we present Claw-R1, an interactive step-level data middleware system for agentic RL. Claw-R1 connects heterogeneous agent runtimes with RL training backends through two core components: a Gateway Server and a Data Pool. The Gateway Server captures multi-turn interaction steps through a unified LLM API entry point, while the Data Pool organizes them into step-level records consisting of prompt IDs, response IDs, rewards and other metadata. In our demo, users can interactively inspect live trajectories, examine the state, action, and reward of each step, curate data by quality and readiness, and configure training-ready batches for different downstream RL algorithms. Overall, Claw-R1 treats agent interaction traces as managed data assets rather than temporary runtime logs. Through this demonstration, we hope to encourage the community to recognize the importance of data management in agentic RL. Our code is available at https://github.com/AgentR1/Claw-R1 and the demonstration video can be found at link https://youtu.be/Pw47dAOw6B0.