ReTAC improves video spatial reasoning by targeted extra computation
ReTAC: Residual Test-Time Agentic Computation for Video Spatial Reasoning
Artificial Intelligence
Summary
Understanding spatial relationships in videos is tricky because some details can be missed or misunderstood by large AI models at first glance. The authors present ReTAC, a method that carefully adds extra computation only when the initial reasoning isn’t confident or correct. Instead of redoing everything, it focuses on the parts that need more attention, like planning steps to find missing clues and double-checking where mistakes might have happened. This approach helps the AI models get better at figuring out spatial info in videos without wasting effort on already solved parts.
What this means in practice
- •For video analytics developers: Integrate selective recomputation methods to enhance video spatial understanding in AI systems without retraining models.
- •For computer vision engineers: Improve spatial reasoning tasks by adding targeted, stage-specific verification and recovery steps to multimodal models processing video data.
Authors
Yike Wu, Keyan Chen, Qixiong Wang, Xi Xiang, Maofei Que, Xiaolong Jiang, Xiaoqin Zhang, Ling Shao, Shijian Lu
Abstract
Test-time agentic computation can improve multimodal large language models (MLLMs) through planning, tool use, iterative perception, and reflection, but indiscriminate computation may duplicate resolved work, introduce noisy evidence, and perturb correct predictions. We propose that such computation should be residual: allocate additional computation only to requirements unresolved by the base model and, after a localized failure, recompute only the affected states. We instantiate this formulation in ReTAC, a training-free multiagent framework for video spatial reasoning. ReTAC organizes supplementary computation around unresolved requirements. It plans ordered steps to acquire missing spatial evidence and combines the observations with compatible native evidence for reasoning. In addition, it uses stage-aligned verification to localize failures to planning, perception, or reasoning. Dependency-based recovery recomputes only affected states while reusing unaffected ones. Extensive experiments show that ReTAC achieves the highest overall score among the compared methods across three mainstream MLLMs and two widely adopted benchmarks. Further analyses show that gains from additional computation depend not only on how much is used, but also on where it is allocated and which valid intermediate states are preserved.