Driving scene model improves reasoning and segmentation accuracy
DSRS: Teaching MLLMs to Perceive, Reason, and Segment in Complex Driving Scenes
Computer Vision and Pattern Recognition
Summary
Understanding complex driving scenes is hard because it involves recognizing many things and their relationships, such as other cars, traffic rules, and risks. The authors introduce a new task called driving scene reasoning segmentation that challenges models to do all this and create detailed visual maps called segmentation masks. They built a special dataset to help train and test these models and designed DSRS-Net, a model that combines images with 3D information for better understanding. Their model performs better than others on this task and related language-guided tasks.
What this means in practice
- •For automated vehicle developers: Improve perception systems by integrating reasoning about spatial relations and traffic rules for safer driving scene segmentation.
- •For robotics engineers: Enhance robot vision in complex environments by combining geometric and linguistic context for better scene understanding.
Authors
Sicen Guo, Zhengfei Song, Kuanhsun Chen, Xu Huang, Ping Zhong, Wei Ye, Shaoyi Du, Rui Fan
Abstract
Existing reasoning segmentation frameworks typically employ multimodal large language models (MLLMs) with broad world knowledge to extract semantics in a human-like manner and generate corresponding segmentation masks. Nevertheless, these frameworks lack the necessary scene-specific awareness. This paper introduces driving scene reasoning segmentation (DSRS), a scene-specific task that requires models to interpret implicit instructions, infer spatial relations, detect potential risks, handle challenging conditions, and comprehend underlying traffic rules, ultimately generating accurate segmentation masks. Meanwhile, we establish a benchmark dataset to facilitate future research on the DSRS task. To pioneer this task, we propose DSRS-Net, a novel architecture with advanced reasoning segmentation capability for automated driving. Specifically, DSRS-Net seamlessly infuses geometric features for deeper reasoning segmentation, without compromising the original multimodal fusion mechanism. This strategy effectively mitigates the modality representation mismatch in MLLMs caused by RGB-only pretraining, while enhancing the integration of geometric, contextual, and linguistic features. Extensive experiments demonstrate the superior performance of DSRS-Net in both driving scene reasoning segmentation and other language-driven segmentation tasks across various public datasets. The source code and dataset will be released upon publication.