Papers for

marine robotics quality assurance teams

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

Search based testing finds vision language model errors in underwater robot software

Search-Based Metamorphic Testing of Vision-Language Models in Autonomous Underwater Robotic Software

Abstract: Our industry partner focuses on quality assurance for industrial systems across multiple domains, including maritime systems, such as overwater vessels and autonomous underwater robots (AURs). Despite the strong performance of vision-language models (VLMs) in scene understanding, image captioning, and object recognition, their use in AUR software operating in underwater environments is underexplored. Therefore, in this context, it is important to evaluate the quality of VLMs for integration into AUR software and, so, automated software testing tools are needed to assess their suitability and improve their dependability. To this end, we propose a search-based metamorphic testing approach (MetaVLM) that identifies a minimal set of transformations on underwater images to induce incorrect model predictions, thereby revealing VLM failures. We employ NSGA-II as a multi-objective search algorithm and evaluate it over open-source VLMs, BLIP and CLIP, against a random search baseline. Results demonstrate the strengths and limitations of each VLM in the context of AUR software systems. Based on the results, we derive lessons for software engineering practitioners and researchers working on quality assurance of VLM-based software systems.

Tue 15 SeptSoftware Engineering
The gist
Underwater robots rely on vision-language models (VLMs) to understand scenes and identify objects, but these models have not been well tested underwater. The authors developed a method called MetaVLM that automatically changes underwater images in small ways to find when these models make mistakes. They used an advanced search technique to efficiently find these image changes and tested two popular models, BLIP and CLIP. Their work shows where these models perform well or fail when used in underwater robot software and offers advice for developers working with such AI systems.
Open → 2609.17007v1