LLM based and rule based fault injections reveal different cloud software errors
Beyond Fixed Fault Models: Comparing LLM-Based and Rule-Based Fault Injection in OpenStack
Software Engineering
Summary
Testing cloud software by deliberately inserting faults helps find hidden bugs. This study compares traditional rule-based fault injection tools with newer AI models known as large language models (LLMs) that generate faults based on context. Both approaches found problems at similar rates in OpenStack software, but the types of errors they caused were different. LLM-induced faults were more likely to cause serious system crashes, while rule-based faults often led to silent or multi-part issues. The authors conclude that while LLMs add useful variety to testing, they don’t completely replace manual methods and need careful control and checking.
Software fault injectionLarge language modelsRule-based fault injectionOpenStackNovaCinderSoftware testingFault modelsCloud computingMutation testing
Authors
Giuseppe De Rosa, Pietro Liguori, Domenico Cotroneo
Abstract
Software Fault Injection (SFI) supports testing of cloud systems by introducing software defects and observing their manifestation. Rule-based injectors such as ProFIPy provide controlled and reproducible source-level mutations but require fault patterns to be encoded manually. Large Language Models (LLMs) offer a data-driven alternative by generating context-dependent software faults. We compare two code LLMs, Qwen2.5-Coder and DeepSeek-Coder, with ProFIPy in OpenStack's Nova and Cinder services. On shared injection targets, activation and observable-failure rates are comparable, but operational profiles differ: LLM-generated faults produce more Catastrophic outcomes on Nova, whereas ProFIPy produces more Silent and Multi-component effects. The sampled LLM outputs also differ in how they manifest failure, while showing greater agreement in their propagation scope. These findings show that LLM-based fault injection extends the behavioral coverage of fixed fault models without establishing general superiority, and that practical adoption still requires controlled generation, runtime validation, system-level oracles, and reproducible experimental provenance.