Grounding Agentic VLMs with Dedicated Segmentation for Fine-Grained Vehicle Damage Assessment
2026-08-03 • Computer Vision and Pattern Recognition
Computer Vision and Pattern RecognitionArtificial Intelligence
AI summaryⓘ
The authors investigate how vision-language models (VLMs) struggle to accurately locate very small vehicle damages like scratches in images, even though they can recognize them well. They find that a top VLM often mistakes reflections for damage and misses fine details. To fix this, they combine the VLM with a specialized segmentation model that better finds tiny damages, improving accuracy and reducing false damage reports. They also introduce new ways to measure performance on detecting small objects and test their combined system in a realistic setting.
vision-language modelsspatial groundingvehicle damage assessmentmulti-task segmentationfocal losssupervised contrastive learningtiny-object detectionLangGraphclass imbalanceperformance metrics
Authors
Vishwajeet Shivaji Hogale, Anjali Pai, Nitya Ravi
Abstract
Vision-language models (VLMs) are increasingly deployed as reasoning agents in real-world visual assessment pipelines, yet their spatial grounding remains unreliable for fine-grained, visually ambiguous targets. We study this gap in the context of automated vehicle damage assessment, where fine-grained defects such as scratches and hairline cracks occupy few pixels, produce weak gradient signal, and are easily confused with reflections and surface texture. We show that a state-of-the-art VLM (Qwen-VL) achieves strong semantic classification accuracy (87.3%) on this task but is systematically ungrounded at the spatial level: it hallucinates damage in reflective regions, misses elongated scratches entirely, and produces spatially inconsistent outputs when prompted for localization. We propose TinyDamage, a hybrid architecture that delegates spatial grounding to a dedicated multi-task segmentation model while reserving the VLM for semantic reasoning and report generation. On the segmentation side, we find that the choice of loss function has an outsized and underexplored effect on tiny-object grounding: focal loss, widely used for class imbalance, collapses tiny-damage detection to zero, while a supervised contrastive objective measurably improves damage/background separability. We integrate the segmentation model into a 7-node LangGraph agent pipeline that grounds every VLM generation step in the segmentation output, and show that this grounding reduces the report hallucination rate from 92% (text-only) and 78% (image-only) to 31% in a controlled evaluation on 100 human-verified reports. We introduce DET_l, a permissive per-category detection metric for evaluating tiny-object grounding under class imbalance, and report latency and reliability characteristics of the deployed pipeline.