HiSTEVE enables more autonomous agents for complex open world games
HiSTEVE 1.0: Harnessing Human-Interface VLA Agents for Open-World Games
Computer Vision and Pattern Recognition
Summary
Open-world games like Minecraft are hard for bots to play the way humans do because they need to plan many steps ahead and control the game interface naturally. The authors developed HiSTEVE, a new system that breaks down tasks into smaller steps and uses a special agent that sees the game and controls the keyboard and mouse directly. HiSTEVE remembers progress, plans tasks carefully, and learns to recover from mistakes, allowing it to successfully complete complex tasks like crafting tools from scratch.
What this means in practice
- •For game ai developers: Create agents that autonomously complete complex tasks in open-ended games using natural game controls without relying on predefined APIs.
- •For robotics software teams: Adapt hierarchical planning and native interface control techniques from HiSTEVE to improve real-world autonomous agents interacting via complex GUI systems.
Authors
Zecong Tang, Zian Guan, Yuncheng Guo, Zhanqiu Zhang, Yiwen Guo, Bo Dai
Abstract
Building virtual embodied agents that operate in open-ended games like human players is a long-standing goal requiring both long-horizon autonomy and reliable native-interface control. As a procedurally generated open world, Minecraft provides a natural testbed for studying the integration of these capabilities. Existing agents typically provide either native visuomotor control without long-horizon autonomy or high-level planning executed through environment APIs or predefined skills. To bridge this gap, we propose HiSTEVE, a hierarchical VLA agent with a capability-aware agent harness. At the orchestration level, we equip the harness with structured memory for tracking progress and a VLM planner for selecting subtasks within the VLA's capabilities. To reliably determine subtask completion, we further introduce a policy-coupled task completion state detector that conditions on VLA policy context to signal candidate completion. At the control level, a single VLA executes all subtasks with vision-only perception and native keyboard-and-mouse actions. To mitigate interference between heterogeneous GUI and non-GUI interactions, we introduce regime-specific RMSNorm branches and separate mouse-movement token embeddings. We also train the VLA through world knowledge injection, imitation learning, and reinforcement learning with environment feedback to improve reliability and recovery. HiSTEVE achieves state-of-the-art atomic-task performance, scoring 85.0% on MineStudio Bench and a 37.4% average success rate on OpenHA, while also completing long-horizon tasks such as crafting an iron pickaxe from scratch like human players.