Environment aware scheduling improves agent reinforcement learning efficiency
EnvSched: Environment-Aware Scheduling for Rollout-Efficient Agent Reinforcement Learning
Machine LearningComputation and Language
Summary
Generating and checking agent learning experiences can be costly, especially when there’s limited time or resources. The authors introduce EnvSched, a system that smartly chooses which tasks and environments to focus on, based on previous results, to learn faster and better. EnvSched balances picking tasks likely to teach the agent something new and ensuring wide coverage of different environment types. In tests, EnvSched reached higher performance more quickly than random task choices across several benchmarks.
What this means in practice
- •For ai system developers: Optimize training of agentic reinforcement learning models by scheduling tasks across diverse environments for more efficient learning.
- •For autonomous software engineers: Improve automated agent testing workflows by selecting environment–task pairs that maximize learning value within limited rollout budgets.
Authors
Sihan Yang, Weichen Bai, Ruichuan An, Yiming Cheng, Zhiyuan Feng, Tianyi Jiang, Renye Yan, Ning Ding, Yifan Zhang, Renrui Zhang, Yehui Tang, Wentao Zhang
Abstract
As executable environments for language agents scale, generating and verifying their rollouts becomes increasingly expensive. Under finite rollout budgets, this raises an important question: can data management at the environment level make agentic reinforcement learning (RL) more efficient by organizing, evaluating, and scheduling task instances together with the environments that determine their tools, states, transitions, and verification? To address this question, we introduce EnvSched (Environment-aware Scheduling), an online scheduler that learns from rollout history. EnvSched assigns scheduling value through two complementary modeling choices: exploitation prioritizes tasks likely to provide informative reward differences, while coverage spreads each batch across underobserved environments and interaction patterns. EnvSched organizes environment--task pairs as nodes in a dual-relation graph, with within-environment edges capturing task semantics and shared tool, state, and verifier structure, and cross-environment edges capturing abstract execution and reward topology. By propagating historical rollout statistics along both edge types, it predicts within-task reward variance. These predictions guide a scheduler that allocates each batch to tasks with high predicted reward variance and underobserved environments and tasks. Environment quotas and a similarity penalty further limit concentration and redundancy, yielding informative and diverse batches without additional pilot rollouts. In RL on Qwen3-8B, EnvSched achieves an average score of 48.81 across BFCL-v3, tau-Bench, and ACEBench-Agent after 40 training steps, exceeding the highest score attained by random task selection after 120 steps. It also improves downstream performance across model sizes at the same training budget.