Diversity guided testing finds more varied failures in large language model apps
Diversity-Guided Search-Based Testing of Large Language Model Applications
Software Engineering
Summary
Large language model applications, like chatbots and voice assistants, can give wrong or harmful replies and are hard to test due to many possible inputs. The authors created a testing method that tries different kinds of inputs while aiming to find many different kinds of errors. Their method keeps track of previous tests and focuses on exploring new input types, sometimes picking very different inputs to find new problems. Testing this on different apps and language models showed it finds more errors and a wider variety of problem types than random testing. They found a balanced strategy worked best to both find more errors and keep diversity in test cases.
What this means in practice
- •For software testers: Run more effective tests on AI based apps by generating varied inputs that can reveal multiple kinds of errors and safety issues.
- •For automotive software teams: Improve reliability of in-vehicle LLM systems by detecting a broader range of failures through guided diverse test inputs.
Authors
Lev Sorokin, Ivan Vasilev, Ken E. Friedl, Andrea Stocco
Abstract
Large Language Model (LLM)-based applications are increasingly deployed across domains including customer service, education, and mobility. These systems are prone to inaccurate, fictitious, or harmful responses, and their vast, high-dimensional input space makes systematic testing particularly challenging. In this paper, we present a search-based testing framework for LLM-based applications that incorporates failure diversity as an explicit optimization objective. Building on a discretization along stylistic, content-related, and perturbation dimensions, our framework maintains an archive of generated tests and rewards distance from that archive, while a repopulation operator periodically replaces non-failing tests in the population to sustain exploration. The repopulation operator is parameterized by its sampling strategy: replacement candidates are drawn either uniformly or by greedy distance maximization, two settings we compare empirically. We evaluate both against several baselines across three case studies---LLM safety, in-car navigation, and in-vehicle function control---covering five systems under test, eight LLMs, and 18 distinct test configurations, with over one million executed tests. Our results show that all guided variants detect substantially more failures than random and combinatorial search in nearly all configurations. Among them, diversified search detects fewer failures, but covers a broader range of failure types in all case studies, with greedy repopulation offering the best tradeoff.