Llm agents learn better search skills from fictional rule based worlds

PhantomEnvironments: Training LLM Agents in Fictional Worlds

Machine LearningArtificial IntelligenceComputation and Language

Summary

Getting AI language models to learn tasks by practicing in the real world is often expensive and complicated. The authors show that you can train these AI agents in completely made-up worlds built from simple, rule-based stories and articles. Even though these fictional environments contain no real facts, agents trained this way get better at searching for answers in the real world, sometimes beating agents trained on actual data. This approach is cheap and helps agents scale their effort based on question difficulty, making it a promising method for teaching AI to think through complex questions.

What this means in practice

  • For ai developers: Train AI search agents in inexpensive fictional environments to improve real-world multi-step question answering.
  • For enterprise search teams: Use agents trained in simulated fictional contexts to enhance multi-hop information retrieval on company data.

Authors

Anmol Kabra, Swathi Saravana Selvam, Albert Gong, Chao Wan, Christian Belardi, Dongyoung Go, Katie Z. Luo, Kilian Q. Weinberger

Abstract

Training LLM agents with reinforcement learning (RL) is bottlenecked by environments, which must provide verifiable rewards, support long-horizon interaction, and scale cheaply. Existing approaches rely on costly human-curated data or on LLM-generated environments that risk hallucinations and benchmark contamination. We show that LLMs can instead be trained into capable search agents using synthetic environments generated entirely by rules, whose generation requires no LLM and has zero marginal cost. We build PhantomEnvironments, multi-turn RL environments from fictional worlds, where agents must search a corpus of templated articles to answer multi-hop questions. Despite sharing no facts with the real world, these strikingly simple environments yield agents that transfer to real-world multi-hop search benchmarks, often outperforming real-world training data on newer benchmarks. Trained agents generalize to unseen fictional universes, and Qwen models learn to scale their search budget roughly linearly with question difficulty, suggesting emergent search scaling from environment interaction alone. Ablating environment complexity reveals that hop count drives transfer more than constraints or comparisons: even the simplest rule-generated environments are a surprisingly effective, free resource for training generalizable LLM agents.