Vision language models systematically tested for safer automotive perception
Towards Systematic Qualification of Vision-Language Models for Automotive Perception Systems
Computer Vision and Pattern RecognitionSoftware Engineering
Summary
Vision-language models (VLMs) that combine images and text can help cars understand their surroundings. However, these models sometimes see things that aren’t really there or miss objects that are present, which can be dangerous for driving. The authors study ways to test these models both when they are being designed and while they are running. They create a method that uses a structured system of labels and synonyms to check how often and what types of mistakes the models make on a standard self-driving car dataset. This helps engineers compare models and make safer choices for car perception systems.
What this means in practice
- •For automotive systems engineers: Assess and compare vision-language models systematically to select safer options for vehicle perception systems before deployment.
- •For autonomous vehicle software teams: Implement a combined design-time and run-time testing workflow to monitor and reduce object detection errors in driving environments.
Tested on one dataset.
Authors
Malsha Ashani Mahawatta Dona, Konstantinos Rokanas, Alexander Säfström, Krishna Ronanki, Christian Berger
Abstract
The field of Artificial Intelligence has been adopted for many application domains. Vision Language Models are one of the recently advanced AI techniques that have been explored to support automotive features such as vehicle perception, and safety assurance. However, such language models are prone to hallucinations, posing a potential threat to the safety of automotive systems that may incorporate them. Within the automotive domain, VLMs could not only hallucinate traffic objects, but could also fail to identify traffic objects that are actually present, which may potentially lead to dangerous situations. Though we have observed a growing body of literature that proposes verification and validation techniques for safe and trustworthy AI, these methods are often studied in isolation, focusing either on run-time or design-time phases. Such isolated techniques could be insufficient in safety-critical, realistic contexts such as automotive perception systems. In this paper, we analyze design-time and run-time verification and validation techniques based on a taxonomy presented by Huang et al. We present an automotive study in which a design-time qualification workflow is proposed to complement run-time monitoring. This workflow combines a fixed safety-relevant ontology-based structured annotation system together with a synonym-based evaluation process to statistically evaluate three state-of-the-art VLMs against data from the nuScenes dataset. We observed that the proposed technique enables deterministic and repeatable quantification of the hallucinations VLMs generate in automotive perception-related tasks. The proposed workflow supports model comparison and deployment-oriented engineering decisions within the design-time verification and validation process and will contribute to a holistic verification strategy that strives towards trustworthy automotive perception systems