LLM fault injection offers different cloud system testing behaviors than rule methods
Beyond Fixed Fault Models: Comparing LLM-Based and Rule-Based Fault Injection in OpenStack
Software Engineering
Summary
Testing cloud software by deliberately adding bugs helps find problems before users do. The authors compared two methods: one uses fixed, manually coded rules to add bugs, and the other uses AI language models to create faults based on the code context. Both methods triggered failures at similar rates but caused different types of problems, showing that AI-generated faults cover different testing scenarios. This means AI can add value without fully replacing traditional testing tools, and practical use needs careful setup for reliability and repeatability.
What this means in practice
- •For cloud platform engineers: Use AI-generated faults alongside rule-based mutants to increase test coverage on cloud services by revealing additional failure behaviors.
- •For software quality assurance teams: Integrate large language model fault injection to complement manual fault patterns and identify different types of silent or catastrophic failures in complex systems.
Authors
Giuseppe De Rosa, Pietro Liguori, Domenico Cotroneo
Abstract
Software Fault Injection (SFI) supports testing of cloud systems by introducing software defects and observing their manifestation. Rule-based injectors such as ProFIPy provide controlled and reproducible source-level mutations but require fault patterns to be encoded manually. Large Language Models (LLMs) offer a data-driven alternative by generating context-dependent software faults. We compare two code LLMs, Qwen2.5-Coder and DeepSeek-Coder, with ProFIPy in OpenStack's Nova and Cinder services. On shared injection targets, activation and observable-failure rates are comparable, but operational profiles differ: LLM-generated faults produce more Catastrophic outcomes on Nova, whereas ProFIPy produces more Silent and Multi-component effects. The sampled LLM outputs also differ in how they manifest failure, while showing greater agreement in their propagation scope. These findings show that LLM-based fault injection extends the behavioral coverage of fixed fault models without establishing general superiority, and that practical adoption still requires controlled generation, runtime validation, system-level oracles, and reproducible experimental provenance.