InfluenceField improves prediction of visual changes in language models
InfluenceField: A Differentiable Field with Interventionally Identifiable Causal Structure for Multimodal World Modeling
Machine Learning
Summary
Multimodal language models that understand images and text together often struggle to predict what happens when small changes are made to parts of an image. The authors introduce InfluenceField, a new method that helps the model understand how these local changes can affect the overall interpretation. InfluenceField works by creating a detailed map of the image features and how influence spreads across them, allowing the model to better predict effects of changes. Their experiments show that this method improves accuracy, especially for questions involving planning or imagining what-if scenarios.
multimodal modelsvisual-linguistic correlationinterventional supervisionlatent fieldcausal structurelanguage modelingcounterfactual reasoningtransition operatorCausalVQAstructural regularization
Authors
Zihao Yang, Zijia Wang, Zhiqiu Huang
Abstract
Multimodal large language models often capture visual-linguistic correlations but struggle to predict how local visual interventions propagate and affect downstream answers. We introduce InfluenceField, an intervention-aware latent field inserted between the visual encoder and language decoder. It lifts patch features into a continuous spatial representation, propagates directed influence over multiple steps, and predicts local intervention effects through a shared transition operator. Training jointly optimizes language modeling, cross-environment invariance, counterfactual rollout supervision, and structural regularization. For a nonlinear finite-basis population model, we show that target-aligned interventional supervision, together with a one-step separation condition on the transition, restricts admissible representations to within-location reparameterizations, so that the directed dependency graph of the full transition is recovered exactly. A linear specialization gives an exact partial-coverage characterization and a finite-loss stability bound, and the field analysis derives the spatial profile of coefficient interventions together with a shared-channel calibration result. On CausalVQA, InfluenceField improves overall accuracy over its backbone by 13.1 percentage points, with the largest gains on the planning and hypothetical categories. Capacity-matched baselines and structural controls attribute the gains in robustness and factual-counterfactual consistency to the causal objectives rather than to added capacity.