Papers for

augmented reality creators

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

Benchmarking counterfactual spatial reasoning with causal transition modeling

CST-Bench: Benchmarking Counterfactual Spatial Transition Reasoning with Causal Transition On-Policy Self-Distillation

Abstract: Spatial reasoning is a fundamental capability of multimodal large language models (MLLMs), yet existing benchmarks mainly focus on static scene understanding. When objects are moved or swapped, models must infer not only spatial relations but also how interventions change the scene state. To address this problem, we introduce CST-Bench, a benchmark for counterfactual spatial reasoning. CST-Bench contains 7,082 examples covering 6 intervention types and 12 transformation operators, with dense causal annotations including intervention sources, affected relations, expected effects, invariant relations, and metric changes. We benchmark general-purpose MLLMs on CST-Bench and further compare post-training strategies on CST-Bench and standard spatial benchmarks. Results show that methods effective for ordinary spatial reasoning can still fail under counterfactual interventions. We further formulate counterfactual spatial reasoning as causal state-transition modeling and propose CT-OPSD, which uses CST-Bench annotations to guide a teacher model in learning counterfactual state transitions and causal reasoning chains. These chains are then distilled into a student model that reasons only from real observations. Overall, experiment results establish CST-Bench as a challenging benchmark and reusable supervision resource for counterfactual spatial reasoning. CT-OPSD further provides an initial demonstration of how its annotations can support model improvement.

Thu 5 NovArtificial Intelligence
The gist
Understanding how objects move and change in space is important but tricky for language models. The authors created CST-Bench, a large set of tests that check if models can reason about changes in scenes when things are moved or swapped. They found that usual methods struggle with these 'what if' scenarios. To help, they designed a new approach called CT-OPSD that teaches models to think about cause and effect in these changing scenes. This work provides both a tough test and a way to make models better at reasoning about space and change.
Open → 2611.07541v1

Spatiotemporal memory system for changing 3D scenes improves reasoning

Gaussian Episodic Memory for Long-Term Spatiotemporal Reasoning

Abstract: Real-world environments undergo continual semi-static changes, requiring a memory representation that captures both the spatial and temporal evolution of a scene. Existing spatio-temporal memories for robots typically rely either on structured representations, which are limited to the entities and attributes captured during construction, or on image-based memories, which are constrained to previously observed viewpoints. We introduce GEMS, a framework that combines a persistent spatio-temporal 3D Gaussian Splatting (3DGS) map with an agentic reasoning pipeline. The 3DGS map is reconstructed once and subsequently updated only in regions where the scene changes, storing only these modifications while preserving visual detail across time. The agentic pipeline leverages this map to render novel views on demand, iteratively accumulating evidence to answer spatio-temporal questions and ground objects from natural-language queries. To advance future research, we introduce GEMS-Bench, a spatiotemporal question-answering benchmark focused on semi-static environments. It comprises 3 real-world and 12 synthetic environments, spanning 473 question-answer pairs. We evaluate GEMS on novel-view synthesis quality across scene updates, 3D object grounding, spatiotemporal question answering, and storage efficiency. GEMS outperforms recent baselines in 3D object grounding and spatiotemporal question answering, while achieving rendering quality on par with dedicated scene-update methods. Code and data: https://abwerby.github.io/GEMS/

Fri 23 OctComputer Vision and Pattern RecognitionRobotics
The gist
Robots and other machines need to remember both how places look and how they change over time to understand their surroundings well. The authors developed GEMS, a system that keeps a detailed 3D memory of scenes and updates only when parts change, saving storage and detail. It also uses this memory to answer questions about objects and events from different viewpoints. They tested GEMS in real and simulated places, showing it performs better than existing methods at recognizing objects and answering questions about scene changes.
Open → 2610.33158v1

Object focused maps update points to handle changes and errors

Point-Level Refinable Semantic Mapping for Spatial Understanding

Abstract: Open-vocabulary 3D semantic maps have emerged as a key spatial memory representation for long-horizon robot autonomy, enabling robots to reason about objects, their spatial relationships, and past observations across time. However, existing object-centric semantic maps are primarily designed for static environments and rely on an append-only point integration strategy, where newly observed points are accumulated but previously integrated points are rarely reevaluated. Consequently, perception errors accumulate over time, and outdated points persist after objects move or disappear. Although recent dynamic mapping methods update object-level representations in changing scenes, they do not explicitly revise noisy or obsolete points within each object. We propose an online refinable semantic mapping framework that continuously refines object representations through probabilistic point-level confidence updating. Each object point maintains a confidence score that is updated online using multi-view temporal evidence: consistent observations reinforce reliable points, while inconsistent segmentation masks, depth measurements, and visibility cues suppress spurious and outdated points. Combined with object tracking and re-identification to preserve instance identities under motion and relocation, our framework enables object-centric semantic maps to be continuously refined rather than monotonically accumulated. Experiments demonstrate improved mapping accuracy over the previous method by 3.46% on Replica and 2.55% on ScanNet, show robustness in dynamic environments and under perception noise, and provide an effective spatial memory for object navigation and embodied question-answering with success rates of 82.4% and 76.7%.

Thu 22 OctRobotics
The gist
Mapping environments for robots usually means saving lots of points that tell where things are. But when objects move or things get seen wrong, old maps keep wrong points forever. The authors created a method that gives each point a confidence score, which changes over time based on new views and checks, helping the robot know which points are still trustworthy. This way, the map gets better, ignoring old or wrong parts as objects move or disappear. Their system works well in tests and helps robots navigate and answer questions about their surroundings.
Open → 2610.31476v1

Hyperclip plus plus fine-tunes clip for better open vocabulary segmentation

HyperCLIP++: Fine-tuning CLIP forOpen-vocabulary Semantic Segmentation in Hyperbolic Space

Abstract: CLIP, a foundational vision-language model, has emerged as a powerful tool for open-vocabulary semantic segmentation. While freezing CLIP's text encoder is known to preserve its generalization capability, recent studies show that fine-tuning both CLIP's text and image encoders jointly significantly enhances segmentation performance, especially for classes from open sets. In this work, we explain this phenomenon from the perspective of hierarchy alignment, since during fine-tuning, the hierarchical level of image embeddings shifts from image-level to pixel-level. We achieve this by leveraging hyperbolic space, which naturally encodes hierarchical structures. Our key observation is that, during fine-tuning, the hyperbolic radius of CLIP's text embeddings decreases, facilitating better alignment with the pixel-level granularity of visual data. Building on this, we propose HyperCLIP++, a novel and parameter-efficient adaptation strategy. HyperCLIP++ directly adjusts the hyperbolic radius of CLIP's embeddings via scaling transformations to achieve a hierarchy alignment to the target task, i.e., segmentation. To ensure this hierarchy alignment is effected consistently across both modalities and preserves their cross-modal alignment during training, HyperCLIP++ integrates a Dual Cross-Relation Communication (DCRC) module that synchronizes these adjustments between the vision and text pathways. Our experiments show that HyperCLIP++ achieves state-of-the-art performance across three benchmarks while fine-tuning only approximately 5% of CLIP's total parameters. More importantly, we observe that after adjustment, CLIP's text embeddings exhibit a relatively fixed hyperbolic radius across datasets, suggesting that the hierarchical level required for this segmentation task might be quantified using the hyperbolic radius.

Mon 21 SeptComputer Vision and Pattern Recognition
The gist
Semantic segmentation means labeling every pixel in an image, which is hard when new classes appear unexpectedly. The authors find that fine-tuning CLIP, a vision-language model, works better when done in hyperbolic space, which helps capture hierarchies. They introduce HyperCLIP++, which adjusts the representation scale to better match detailed pixel-level image data and keeps text and image parts in sync. This approach improves segmentation accuracy while only changing a small fraction of the model’s parameters.
Open → 2609.24564v1

Model articulation of 3D objects from limited sparse views

FAMOS: Feed-Forward 3D Articulation Modeling from Sparse Observations

Abstract: Modeling articulated objects from sparse monocular views is challenging because each observation reveals only partial geometry and motion evidence. Most feed-forward methods infer articulation from a single observation and therefore rely heavily on learned category-level shape priors. We present FAMOS, a feed-forward model that predicts movable-part segmentation and joint parameters from a sparse, unordered set of partial point clouds. Our model jointly reasons over multiple observations and naturally supports a variable number of inputs, including a single view. To aggregate articulation cues across observations, we introduce a Multi-state Articulation Transformer with alternating state-wise and global attention. We further propose an observed articulation span objective that supervises the motion range each part exhibits across the input observations, encouraging the model to leverage the full observation set. To overcome the limited scale and diversity of existing datasets, we introduce a procedural data generator that synthesizes self-annotated assets during training. Experiments on PartNet-Mobility, ACD, and ArtiCraft-10K demonstrate consistent improvements over both feed-forward and optimization-based baselines. Project page: https://kevinqu7.github.io/famos

Thu 17 SeptComputer Vision and Pattern RecognitionArtificial IntelligenceRobotics
The gist
Figuring out how parts of an object move based on just a few pictures is hard because each image only shows a small piece of the story. The authors created FAMOS, a method that looks at several partial views together to better understand how parts of objects can move. They use a special kind of neural network called a Multi-state Articulation Transformer to combine information across images. Their approach also uses a new training trick to learn the full range of motion from available views. Tests show that FAMOS works better than previous methods.
Open → 2609.20817v1