New survival sandbox benchmarks sustained goal execution in autonomous agents
ASIWorld: Benchmarking Autonomous Agents in a Persistent Survival Sandbox
Artificial Intelligence
Summary
Testing autonomous agents involves seeing if they can carry out long and complex goals despite limited information and changing conditions. The authors created ASIWorld, a survival-style game environment where agents manage resources, explore, and fight to survive while working on tasks. They studied how different agent setups completed these challenges, observing not just success but also how they acted and the costs involved. This helps show that success depends on both what agents do and how they do it over time.
What this means in practice
- •For robotic system developers: Evaluate and improve long-term decision-making in robots by testing them in a persistent survival sandbox environment introduced by ASIWorld.
- •For game ai developers: Use ASIWorld to benchmark AI strategies that require coordinating multiple interdependent actions under uncertainty and delayed effects.
Authors
Jianfeng Lin, Runze Wang, Haorong Hong, Yuxuan Xie, Zeno Zeng, Zhongyuan Liu
Abstract
Evaluating autonomous agents requires examining whether they can translate goals into coherent action across extended interactions. In persistent environments, this requires coordinating interdependent actions under incomplete observations, accounting for delayed effects, and adapting as earlier decisions shape subsequent opportunities. We introduce ASIWorld, a survival sandbox that combines persistent world and character state with character-scoped observations and a programmable asynchronous interface to study sustained goal execution. Agents pursue interdependent production and exploration tasks through direct actions or agent-authored programs that continue running between model calls. The evaluation assesses task success, the attainment of development achievements, and survival costs, complemented by an analysis of execution behavior. In an exploratory study of nine agent configurations using a common harness, eight complete the production task and six complete both tasks. Analyses of execution trajectories and performance across difficulty levels reveal different preparation, combat, and recovery behaviors among configurations with similar task outcomes. Incomplete preparation is associated with stalled progression, while configurations that complete the tasks differ in survival costs and the fulfillment of additional achievement criteria. These findings illustrate the value of assessing sustained goal execution through both task outcomes and the behaviors and costs associated with achieving them.