Benchmarking counterfactual spatial reasoning with causal transition modeling

CST-Bench: Benchmarking Counterfactual Spatial Transition Reasoning with Causal Transition On-Policy Self-Distillation

Artificial Intelligence

Summary

Understanding how objects move and change in space is important but tricky for language models. The authors created CST-Bench, a large set of tests that check if models can reason about changes in scenes when things are moved or swapped. They found that usual methods struggle with these 'what if' scenarios. To help, they designed a new approach called CT-OPSD that teaches models to think about cause and effect in these changing scenes. This work provides both a tough test and a way to make models better at reasoning about space and change.

What this means in practice

  • For ai application developers: Improve AI systems that understand and predict spatial changes in dynamic environments by training on counterfactual scenarios.
  • For augmented reality creators: Build augmented reality experiences that accurately respond to user-induced object movements by leveraging models trained on counterfactual spatial changes.

Authors

Ying Li, Yongle Cheng, Jing Li, Xingquan Cai, Yawei Luo

Abstract

Spatial reasoning is a fundamental capability of multimodal large language models (MLLMs), yet existing benchmarks mainly focus on static scene understanding. When objects are moved or swapped, models must infer not only spatial relations but also how interventions change the scene state. To address this problem, we introduce CST-Bench, a benchmark for counterfactual spatial reasoning. CST-Bench contains 7,082 examples covering 6 intervention types and 12 transformation operators, with dense causal annotations including intervention sources, affected relations, expected effects, invariant relations, and metric changes. We benchmark general-purpose MLLMs on CST-Bench and further compare post-training strategies on CST-Bench and standard spatial benchmarks. Results show that methods effective for ordinary spatial reasoning can still fail under counterfactual interventions. We further formulate counterfactual spatial reasoning as causal state-transition modeling and propose CT-OPSD, which uses CST-Bench annotations to guide a teacher model in learning counterfactual state transitions and causal reasoning chains. These chains are then distilled into a student model that reasons only from real observations. Overall, experiment results establish CST-Bench as a challenging benchmark and reusable supervision resource for counterfactual spatial reasoning. CT-OPSD further provides an initial demonstration of how its annotations can support model improvement.