What to Test Next: Interpretable Coverage Gap Discovery in Driving VLMs
2026-06-01 • Computer Vision and Pattern Recognition
Computer Vision and Pattern RecognitionSoftware Engineering
AI summaryⓘ
The authors address the problem of testing driving vision-language models (VLMs) across different driving conditions, many of which have not been well tested yet. They introduce SliceScorer, a straightforward and safe way to identify these under-tested scenarios by combining a preference for rare cases and checking conditions similar to ones where failures happened. They then build SliceNav, a system that uses language models to create testing workflows based on developer questions, helping find risky gaps more effectively. Their experiments show this approach works better than previous methods and produces diverse testing suggestions.
vision-language modelsOperational Design Domain (ODD)slice testingstress testingscoring ruleexposure-based coverageneighbor-failure priorlarge language modelsverification pipelinesafety validation
Authors
Abhishek Aich, Sparsh Garg, Vijay Kumar BG, Turgun Yusuf Kashgari, Manmohan Chandraker
Abstract
Driving vision-language models (VLMs) must accurately understand scenes across diverse conditions defined by Operational Design Domains (ODDs), yet verification remains sparse: many slices are missing, making empirical failure rates unreliable. We propose SliceScorer, a deterministic scoring rule for missing-slice recommendation that combines (i) an exposure-based coverage prior to prioritize rare, under-tested regions, and (ii) a neighbor-failure prior that propagates risk from similar tested conditions. SliceScorer is deliberately simple - interpretable, auditable, and conservative - properties essential for safety-critical validation. For stress testing beyond the declared ODD, we embed SliceScorer within SliceNav, an LLM-orchestrated verification pipeline where the model interprets developer queries to select relevant operators (triage, scoring, acquisition, evaluation) and vocabulary extensions, composing verification workflows while keeping all scoring deterministic and auditable. Experiments on three driving VLMs (WiseAD, DriveMM, Cosmos-Reason2-2B) show that SliceNav surfaces high-risk coverage gaps more effectively than prior slice-discovery methods while maintaining diverse recommendations across the condition space. Ablations confirm both scoring components contribute, and qualitative analysis demonstrates end-to-end workflows from developer query to targeted evaluation.