SpaceAct evaluates 3D reasoning and actions in complex puzzles
SpaceAct: Benchmarking 3D Spatial and Temporal Reasoning via Closed-Loop Action and Active Perception
RoboticsArtificial Intelligence
Summary
To do tasks in 3D spaces, computers need to understand shapes, movements, and how to act based on what they see. The authors created SpaceAct, a test that uses puzzles made of bricks and mechanical parts to see how well models can look around, plan, and perform actions while avoiding mistakes like collisions. They found that even the best models struggle with understanding 3D scenes and refining actions, especially when they can't see everything clearly or have to adjust plans on the fly. This benchmark helps show where improvements are needed in teaching AI to interact with the physical world.
What this means in practice
- •For robotic system developers: Test and improve robot perception and manipulation skills in complex 3D tasks involving occlusion and precise movement using SpaceAct’s interactive puzzles and metrics.
- •For augmented reality engineers: Use SpaceAct to benchmark and develop systems that align virtual objects in real 3D spaces with accurate positioning and interaction feedback.
Authors
Chenglong Shi, Zekai Yuan, Yue Song, Jon Pu, Will Wu
Abstract
To solve problems in 3D space rather than merely describe images, multimodal large language models (MLLMs) need 3D spatiotemporal reasoning: reconstructing 3D geometry from occluded 2D views, anticipating physical dynamics, and executing precise, closed-loop actions. Prior studies test passive VQA on static images or symbolic planning, requiring no execution or supplying privileged coordinates while ignoring collision, statics, and timing. We introduce SpaceAct, an interactive benchmark evaluating 3D spatiotemporal reasoning via closed-loop physical action and active perception, with SAPA, a Spatiotemporal Active-Probing Agent framework. Built on interlocking bricks and mechanical puzzles, it has 108 verified instances across 36 task families in six categories, without privileged coordinates, and diagnostic modes (+plan, +state, no-camera, blind, L7 levers). In brick-bridging or aperture-threading, models rotate the camera to resolve occlusion, read in-scene rulers, output 6-DoF poses under swept-collision and statics checks, and self-correct via split-screen rollback feedback (dynamic tasks freeze time during deliberation and give five strobe frames). Across 10 frontier MLLMs on the 44 hardest instances, scores span 4.8-79.1 (seven solve at most 27.3% of 6-DoF tasks); for the top three, a reference plan (+plan) closes 44% of the gap while leaving collisions at 5.2 per episode, whereas adding exact 3D state (+state) closes 90% and cuts collisions to 0.5; for four others, +plan lifts the mean score from 10.9 to 35.3 and +state to 81.8; and 46% of failures are collision or observation loops. SpaceAct and SAPA advance evaluation to closed-loop physical interaction, showing that even with a valid plan, MLLMs remain bottlenecked by 3D visual grounding and action refinement, while offering a testbed for active multi-view exploration and explicit 3D grounding.