Minecraft benchmark tests lasting agent skill improvement and transfer
MC-EvoBench: Auditing Experience-Driven Self-Improvement in Minecraft Agents
RoboticsArtificial Intelligence
Summary
It is hard to tell if a Minecraft-playing AI really learns skills it can use again in new games or simply remembers past items or worlds. The authors built MC-EvoBench, a Minecraft test setup that carefully resets the game world and only lets the AI bring declared saved changes from one challenge to another. Their tests show that some AI with advanced planning improves more than simpler ones, but past experience rarely helps consistently on new tasks. Early challenges like collecting and crafting wood can block the observation of later skills. This benchmark helps measure if Minecraft agents truly improve in ways that last and transfer.
What this means in practice
- •For game ai developers: Use MC-EvoBench to assess if game agents can develop skills that persist and transfer to new Minecraft tasks beyond simple memory.
- •For reinforcement learning engineers: Benchmark embodied agents on their ability to convert interaction history into reusable skills across related and unrelated Minecraft challenges.
Authors
Jingwei Song, Zhisheng Chen, Bingfan Zeng, Jing Tang, Yubo Hou, Jinhao Jing, Zhengwei Xie, Linfeng Zhang
Abstract
A central challenge in embodied recursive self-improvement is enabling agents to convert interaction history into capabilities that persist beyond one world and transfer to later tasks. A saved memory or successful final action does not by itself establish that a reusable capability was acquired; evaluation must separate capability change from carried-over resources, unequal source exposure, and early execution failures. We construct MC-EvoBench as a controlled Minecraft benchmark with a fixed catalog of 19 registered evolution chains, including 12 primary evaluation chains and seven calibration or probe chains, together with three graph-aligned paired probe families for transfer. The benchmark represents each task as a task-state graph, defines stage-level environment verifiers, assigns matched source and target roles, resets the target world, and permits only a declared agent artifact to cross the source-target boundary. For each probe, no-prior, related, unrelated, and shuffled source conditions are evaluated under matched environment-step budgets and five seeds; we report TSR, WGP, and VMP and retain provenance records. The benchmark reveals a consistent direct-capability gap between controller and planner-backed variants, while related experience yields no stable transfer: it does not improve either tool probe, its small wood-species gain hinges on one seed, and all intervals include zero. Stage audits further show that early wood-collection and crafting bottlenecks often leave later capabilities unobserved. MC-EvoBench thus offers a controlled testbed for durable, transferable embodied capabilities. Code and data are available at https://github.com/xzw-ustc/mc-bench-open-source