Verifiable agentic environments improve long horizon reinforcement learning
Verifiable Hidden Dynamics Play: Generating Agentic RL Environments from Solved Mechanisms
Artificial IntelligenceComputation and Language
Summary
Training language-model agents to handle long, complex tasks is hard because you need many varied environments and clear signals about success. The authors developed VHD-Play, which creates environments by first solving mathematical models, then turning them into interactive setups agents can use. This method produces thousands of diverse and verifiable environments cheaply and helps train agents that perform much better on a range of tasks, including e-commerce and travel planning challenges. The results show that learning to interact with dynamic states is more important than just solving static problems.
What this means in practice
- •For machine learning engineers: Train language-model agents on diverse, verified environments to improve performance on long-horizon decision-making tasks.
- •For e-commerce platform developers: Use verifiable agentic training environments to build models that reliably manage multi-step commerce simulations without bankruptcy.$Commercial implications: Enables commercial AI systems for managing complex e-commerce operations with fewer failures by ensuring valid learning scenarios.
Authors
Xinjie Shen, Wei Fan, Xudong Guo, Jianhong Tu, Yang Su, Chuqiao Kuang, Yinger Zhang, Dayiheng Liu
Abstract
Language-model agents increasingly face long-horizon tasks with evolving state, interdependent decisions, and delayed outcomes. Scaling their training requires diverse agentic environments, dependable outcome signals, and low extension cost. Existing generation pipelines commonly construct an environment before defining its outcome rule or annotating its trajectories, leaving dynamics and evaluation to be aligned post hoc. VHD-Play reverses this dependency by sampling and solving a mathematical model before a corpus-grounded setter renders its decision process as stateful tools. The executable dynamics and trajectory-scoring reference are inherited from the same solved model. The pipeline produces 3,300 diverse agentic environments at a cost of a few cents each. Training Qwen3.6-35B-A3B on three families raises its mean agentic score from 0.204 to 0.815 in a five-family diagnostic. Gains also appear on held-out instances from all three training families and eight unseen mechanism families, then extend beyond the generated substrate to external benchmarks for general function calling, travel planning, and 365-day e-commerce. On E-Commerce Bench, the trained checkpoint completes every run without bankruptcy and exceeds Qwen3.7-Max. We compare written-out problems with stateful versions that reveal or hide their parameters. The comparison shows that most of the learnable gap lies in stateful interaction rather than underlying problem solving. A frozen 35B setter realizes larger environments, and scale-matched training retains gains as mechanism size and horizon grow, indicating the potential for an evolving training substrate.