Summary
Multimodal AI models that work with both images and language often find it hard to understand how making changes to specific parts of an image affects answers about the scene. The authors propose InfluenceField, a new component that sits between the image understanding part and the language generation part, capturing how changes at one point in an image influence others. This method learns to predict the effects of local changes by modeling directed influence across space and time, improving accuracy especially on complex tasks like planning or imagining what-if scenarios. Their approach also provides a way to identify causal relationships within the model's internal representations.
What this means in practice
- •For ai system developers: Build vision-language systems that better predict effects of local image changes on language outputs for reasoning and planning tasks.
- •For robotics engineers: Use improved causal modeling of visual environments to enhance robots’ understanding of how local actions affect downstream task outcomes.
Abstract
Multimodal large language models often capture visual-linguistic correlations but struggle to predict how local visual interventions propagate and affect downstream answers. We introduce InfluenceField, an intervention-aware latent field inserted between the visual encoder and language decoder. It lifts patch features into a continuous spatial representation, propagates directed influence over multiple steps, and predicts local intervention effects through a shared transition operator. Training jointly optimizes language modeling, cross-environment invariance, counterfactual rollout supervision, and structural regularization. For a nonlinear finite-basis population model, we show that target-aligned interventional supervision, together with a one-step separation condition on the transition, restricts admissible representations to within-location reparameterizations, so that the directed dependency graph of the full transition is recovered exactly. A linear specialization gives an exact partial-coverage characterization and a finite-loss stability bound, and the field analysis derives the spatial profile of coefficient interventions together with a shared-channel calibration result. On CausalVQA, InfluenceField improves overall accuracy over its backbone by 13.1 percentage points, with the largest gains on the planning and hypothetical categories. Capacity-matched baselines and structural controls attribute the gains in robustness and factual-counterfactual consistency to the causal objectives rather than to added capacity.