Explorer-Zero improves long task solving with self-play rewards
Explorer-Zero: Improving Self-Play for Long-Horizon Tasks with Decomposable Verifiable Rewards
Artificial IntelligenceMachine Learning
Summary
Solving long and complex tasks with AI agents usually needs a lot of human help to check if the task is done correctly. The authors created Explorer-Zero, a system where an AI proposes tasks for itself and rewards itself based on partial progress, not just success or failure. This approach helps the AI learn better by breaking down tasks and verifying steps automatically. Their tests show Explorer-Zero works better than other self-play methods and gets close to methods that rely on human-annotated tasks.
What this means in practice
- •For interactive ai developers: Train AI agents to complete lengthy, multi-step tasks without needing human-labeled training data by generating verifiable subtask rewards.
- •For software automation teams: Create automated systems that learn complex workflows by self-generating and verifying progress checkpoints during training.
Authors
Tianyu Liu, Davide Belli, Bence Major, Arash Behboodi, Tribhuvanesh Orekondy
Abstract
Training LLM agents on long-horizon interactive tasks with reinforcement learning typically relies on human-annotated tasks and reliable verification, both expensive to construct for stateful environments. Self-play offers an alternative, but must generate achievable, compositional tasks and provide feedback beyond binary outcomes. We propose Explorer-Zero, a self-play framework that jointly generates tasks and fine-grained, verifiable rewards through environment interaction. A learned proposer explores the environment, constructs tasks, and records intermediate and final states as verification targets. These checkpoints reward partial completion, while alternating proposer--solver training adapts the generated curriculum. We show theoretically that subtask rewards reduce collapse among unsuccessful rollouts when they complete different numbers of subtasks. Without human-annotated training tasks, Explorer-Zero outperforms the evaluated self-play baselines across ALFWorld, ScienceWorld, and AppWorld; on AppWorld, our 30B agent reaches 59.67% task success, approaching 62.65% for RL on human-annotated tasks. Further analyses support task-relevant composition and benefits on complex tasks, while self-play initialization improves subsequent human-data RL across all three benchmarks.