DRScaffold: Boosting Dense-Scene Reasoning in Lightweight Vision Language Models
2026-05-25 • Computer Vision and Pattern Recognition
Computer Vision and Pattern RecognitionArtificial Intelligence
AI summaryⓘ
The authors point out that small vision-language models do okay on simple tasks but struggle when they need to understand complicated scenes with many objects and relationships. To help, they created a new benchmark called DRBench with lots of detailed questions about images, and a training method named DRScaffold that teaches models to reason step-by-step and link their thoughts clearly to the images. Their method improved the models' abilities on these complex tasks without making the models bigger. In fact, a smaller model trained with DRScaffold outperformed a much larger one, showing that better training can replace model size for complicated scene understanding.
vision-language modelsdense-scene reasoningmulti-step inferencegrounded reasoningbenchmarkfine-tuningsupervised learningDRBenchDRScaffoldmodel scale
Authors
Xinrui Shi, Kai Liu, Ziqing Zhang, Jianze Li, Anqi Li, Yulun Zhang
Abstract
Lightweight vision-language models perform competitively on standard benchmarks yet fail systematically in dense-scene reasoning, where multiple objects, attributes, and relations must be jointly grounded and resolved through multi-step inference. Such capability is critical for real-world applications where models must reliably interpret cluttered environments. Yet existing training signals provide no explicit grounding between reasoning steps and the underlying visual entities and relations, leaving lightweight models free to generate fluent but visually unanchored reasoning chains. To address this gap, we first introduce DRBench, a benchmark of 14,573 questions across 2,943 images, organized into five task categories spanning three progressive reasoning layers. Building on DRBench, we propose DRScaffold, a supervised fine-tuning framework that decomposes the supervision target into four causally ordered stages, enforcing grounded reasoning without architectural modification. Experiments on three lightweight VLMs demonstrate substantial gains on DRBench while preserving or improving performance on general-purpose benchmarks. Notably, Qwen2.5-VL-3B trained with DRScaffold surpasses the frozen Qwen2.5-VL-32B on DRBench, demonstrating that structured supervision can substitute for a significant portion of model scale in dense-scene reasoning. Our code and models are available at https://github.com/irene-shi/DRScaffold .