Vision language models struggle to judge shadows without image generation help
Where Did the Light Come From? Diagnosing Cast-Shadow Reasoning in VLMs and Borrowing the Missing Shadow Prior from Image Generation Models
Computer Vision and Pattern Recognition
Summary
Shadows in pictures come from light hitting objects, and understanding them can help computers see better. The authors made a test called ShadowBench to see if vision-language models (that look at images and text) can tell shadow directions, guess missing shadows, or spot shadows that don't fit the scene. They found that many models can tell shadow direction when it's visible but can't guess missing shadows well or judge if shadows are correct. However, models that combine with image generation techniques do better, suggesting that teaching models about shadows might come from how images are created.
What this means in practice
- •For computer vision engineers: Improve AI systems that interpret images by integrating image generation insights about shadows to better assess lighting and physical realism.
- •For graphics software developers: Enhance rendering and editing tools by enabling them to infer correct shadow placements in images where shadows are missing or inconsistent.
Authors
Ryugo Morita, Shunsuke Kitada, Stanislav Frolov, Brian Bernhard Moser, Ko Watanabe, Riku Takahashi, Jinjia Zhou, Andreas Dengel
Abstract
Shadows are observable consequences of latent physical causes, arising from the interaction between illumination, object geometry, and scene structure. A model that understands shadows should be able not only to read a shadow's direction, but also to infer where a shadow should fall when it is absent, and to judge whether a shadow is physically consistent with the scene. We introduce ShadowBench, built from real photographs, which poses these three questions to vision-language models (VLMs). From each scene, we build a correct-shadow, a shadow-removed, and a human-annotated wrong-direction state. Evaluating 19 open VLMs (2B to 78B) and frontier proprietary models, we find that VLMs read a visible shadow's direction well above chance and improve with scale, yet infer the absent direction only weakly and fall to chance at judging physical consistency even under a bias-free two-alternative test that removes the yes-bias of single-image naturalness questions. Only Gemini-3.1-pro exceeds chance on validity, and even then only modestly and far below humans. Chain-of-Thought (CoT) does not help. The missing ingredient is not reasoning but knowledge that image generation models (IGMs) already hold. Routing a VLM through an IGM lifts its inference well above its direct attempt and helps it flag the wrong-direction shadows it otherwise accepts. Cast-shadow direction is a single facet of illumination understanding, but the sharp gap it reveals between VLMs and IGMs suggests image-generation or reconstruction objectives as a route toward illumination-aware VLMs.