Foundation models improve driving detectors in fog and sunlight
Real-World Perception for Autonomous Driving in Adverse Weather: Enhancing Standard Detectors via Foundation-Guided Auto-Annotation
Computer Vision and Pattern Recognition
Summary
Detecting objects around self-driving cars becomes harder in bad weather or tricky lighting. The authors tested several powerful new models to see which works best in real-world conditions. They used the best one to create automatic labels for thousands of driving images where no labels existed. Then, they improved a common real-time detector by training it on these new labels, making it much better at spotting objects in challenging weather. This approach helps standard detectors work better without changing their design or needing lots of manual labeling.
What this means in practice
- •For autonomous vehicle engineers: Improve object detectors to maintain accuracy in fog, direct sunlight, and varying road conditions without redesigning models or extensive manual labeling.
- •For traffic monitoring teams: Use foundation model-generated labels to refine traffic camera object detectors, enhancing reliability under diverse weather and lighting scenarios.
Authors
Sepideh Gohari, Goodarz Mehr, Azim Eskandarian
Abstract
Standard deployment-ready object detectors for autonomous vehicles degrade in adverse weather and lighting conditions without being trained on extensive domain-specific data. While large-scale vision foundation models offer robust zero-shot generalization, their high computational cost makes them impractical for real-time deployment. To bridge this gap, we propose a foundation-guided auto-annotation pipeline that enhances standard detectors without architectural changes. We first benchmark three distinct models, YOLOv8, Co-DETR, and SAM3, on our custom real-world driving dataset spanning 25 unique operational scenarios across various route, weather, and lighting conditions. Based on our analysis, SAM3 demonstrates superior accuracy and resilience across all scenarios. Thus, we deploy it as an offline auto-annotator to generate pseudo-labels on the unannotated subset of our dataset. Fine-tuning the baseline YOLOv8 on these annotations yields a 16.04% higher overall mean Average Precision (mAP) and improves cross-environmental stability compared to the baseline model, highlighted by a 32.73% and 28.65% mAP increase in Residential Direct Sunlight and Highway Fog, respectively. These results demonstrate that standard detectors can achieve environmental resilience without the need for extensive manual annotation or architectural modifications.