WorldFlow improves visual reasoning about hidden scene parts
Seeing Parts, Reasoning about Worlds: Visual Inference under Partial Observation
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
Sometimes when we look at a scene, many things are hidden behind other objects, so we don't know the full picture. This paper helps computers guess what could be hidden based on partial views and helps confirm which guesses are likely or unlikely as new views come in. The authors made a large dataset called WorldScope to train and test this kind of reasoning. They also created a system called WorldFlow to combine evidence from several images and better decide what is actually there. Their system is much more accurate at answering questions about the hidden parts than previous methods.
What this means in practice
- •For robotics developers: Improve robot perception by reasoning about unseen objects using multiple partial views to better understand cluttered environments.
- •For augmented reality engineers: Enhance AR experiences by inferring hidden scene elements from different viewpoints to provide consistent and realistic overlays.
Authors
Wei Wang, Wenqiao Zhang, Yutong Lin, Jun Xiao, Yueting Zhuang
Abstract
World modeling under partial observation requires reasoning about the complete worlds that remain compatible with limited visual evidence. Occluded objects and unseen regions can leave several world states possible; additional views can exclude alternatives and strengthen the conclusions supported by the observations. We introduce WorldScope to study this process through possible-world semantics, evidence-grounded data, and learned visual representations. WorldScope-1.2M provides 1.2 million English question-answer pairs spanning eight world properties, ten task interfaces, and three observation protocols. Its answers encode confirmed facts, supported bounds, and unresolved possibilities. Complementary supervision comprises 4,800 certified counterworld groups with equivalent base observations and different hidden object configurations and query answers. These groups provide physical witnesses of ambiguity and training-only labels for world compatibility and view-induced exclusions. We propose WorldFlow, which composes cross-view entity evidence and support-surface coverage into an image-subset evidence lattice. Counterworld compatibility and transition objectives train subset representations to reflect how new observations constrain possible worlds. A shared answer generator uses these representations to predict the strongest supported conclusion. WorldScope-Bench evaluates claim judgments and evidence-dependent conclusions as views are selected, combined, removed, or ordered. On its 5,000-question test set, WorldFlow reaches 64.34% exact accuracy, improving over the same backbone trained on QA alone by 24.88 percentage points. It retains 50.43% accuracy on the 3,000 questions from structure-disjoint scenes.