Multimodal models struggle to understand physical fields visually
PhysFieldBench: Can Multimodal Models Understand Physical Fields?
Artificial Intelligence
Summary
Multimodal language models, which use images and text, struggle to understand visual information about physical fields like heat or magnetic forces. The authors created PhysFieldBench, a set of tasks to test these models on recognizing and reasoning about different physical fields from images. They found that current models perform near guessing level, but specialized training can improve performance. Their analysis shows many errors come from failing to see patterns or correctly interpret images as physical phenomena. They also tried different training methods to help models generalize better to unseen tasks.
What this means in practice
- •For engineering simulation teams: Use the benchmark to evaluate and improve AI models that interpret simulation output images for engineering physics tasks.
- •For industrial automation developers: Develop AI agents that visually analyze physical field data from sensors to support automated control in manufacturing.
Authors
Yuezhou Ma, Huikun Weng, Jialong Wu, Chenyi Zhao, Hang Zhou, Haonan Shangguan, Jianmin Wang, Mingsheng Long
Abstract
Multimodal large language models (MLLMs) are increasingly envisioned as core components of scientific and engineering agents, yet their ability to interpret physical fields remains poorly understood. Existing physics benchmarks largely emphasize textbook problem solving or intuitive physical reasoning, leaving open whether MLLMs can infer physically meaningful information from continuous field observations. We introduce PhysFieldBench, a benchmark comprising 24 tasks and 1,160 evaluation examples across controlled equation fields, simulated physical fields, and observed physical fields. The tasks assess three forms of inference: identifying physical mechanisms, comparing latent control variables, and predicting outcome properties. Across representative open-source and proprietary MLLMs, zero-shot performance is low: the best model achieves a chance-normalized score of 29.3, while several open-source models remain near chance. In contrast, a task-specific supervised vision transformer performs substantially better, demonstrating that the inputs contain learnable physical information. To diagnose these failures, a structured self-explanation analysis attributes most errors to missed visual patterns and incorrect visual-to-physical mappings. Further, to explore whether post-training can improve physical inference and generalize to unseen tasks, we compare supervised fine-tuning with final answers or chain-of-thought supervision and reinforcement learning. Final-answer supervision performs best overall but transfers less effectively, whereas reinforcement learning after chain-of-thought supervision achieves the best generalization. Together, these findings highlight the need to improve visual-to-physical grounding and cross-task generalization for MLLMs to reliably interpret physical fields in scientific and engineering workflows.