World action models struggle to pick best actions from imagined futures
Beyond Visual Quality: A Study of Test-Time Planning with World Action Models
Robotics
Summary
Planning by imagining different actions and their outcomes sounds helpful, but how to choose the best action using these imagined futures is unclear. The authors studied this by trying to pick the best action from sets of imagined possibilities and found that simple measures like visual prediction quality do not reliably indicate the best choice. They discovered that only a few early decisions show distinct differences that help pick better actions. Their work suggests the value of imagined outcomes should be judged by how well they support decisions, not just by how realistic they look.
What this means in practice
- •For robotics engineers: Improve robot action planning by focusing on evaluating predicted outcomes based on decision usefulness rather than visual realism.
- •For game ai developers: Enhance AI planning in games by designing better scoring methods that identify the most impactful predictions from imagined actions.
Authors
Jianhao Yuan, Yu Yuan, Benjamin Ramtoula, Lukas Vierling, Paul Newman, Lars Kunze, Philip Torr, Daniele De Martini
Abstract
World action models generate actions together with visual predictions of their consequences. These paired outputs create the potential for planning by sampling multiple actions from one state, comparing their imagined outcomes, and choosing the action with the most promising predicted outcome. However, how to use imagined futures to guide action selection remains unclear. We examine this planning potential empirically. First, we estimate an oracle upper bound on selection by choosing the sampled candidate whose realised outcome is best. In a controlled same-state analysis, this choice raises success from 68.9% under uniform random selection to 79.2%. We then test selectors based on visual quality, physical consistency, and task progression as controlled interventions. Some tested selectors yield higher observed success, but the gains are uneven and the matched selectors leave much of the measured opportunity unrecovered. To investigate this gap, we examine whether sampled actions lead to different outcomes, whether these differences are visible in the predictions, and whether a score recognises them. Counterfactual branching from the same states shows that selection opportunity is concentrated in relatively few decisions in the initial candidate sets. Action spread and outcome coverage need not increase together. In a further evaluation across trajectory phases with complete action execution, the tested scores again recover little of the available improvement despite a small gain from learned value. These findings distinguish producing consequential action choices from recognising them in generated futures, motivating the evaluation of WAM predictions through their usefulness for decisions rather than visual quality alone.