AI fault benchmark links real bugs to reproducible fixes and data
AIFaultBench: A Reproducible Benchmark of Real-World Faults in AI Software Systems
Software Engineering
Summary
AI software is hard to fix because bugs can involve code, data, models, and hardware all at once. The authors gathered 770 bugs from many AI projects and made it easy to reproduce and study them by sharing detailed setups and scripts. They managed to recreate about 85% of these bugs, providing a real-world tool to understand and debug AI problems better. This helps improve debugging tools and teaches people about AI software issues.
What this means in practice
- •For ai software engineers: Use documented real bugs and reproduction scripts to diagnose and fix AI system failures more effectively.
- •For ai infrastructure teams: Validate AI deployment stability by testing model and code interactions against a wide range of realistic fault scenarios.
Authors
Mehil B Shah, Mohammad Masudur Rahman, Foutse Khomh
Abstract
AI-based software systems remain difficult to debug because their failures can involve interactions among code, models, data, dependencies, hardware, and execution environments. Existing benchmarks cover specific classes of AI systems, while concerns about fault realism and reproducibility highlight the need for documented execution conditions and evidence. We present AIFaultBench, a benchmark of 770 real-world bugs from 105 AI-based software repositories across 76 organizations. It covers six classes of AI systems: machine learning, deep learning, reinforcement learning, systems based on large language models, agentic AI, and AI tooling and infrastructure. Each bug record connects the original report and affected codebase to structured metadata, environment setup scripts, a reproduction trigger, and investigation evidence. We reproduced 652 bugs (84.7%) under the recorded execution conditions and documented the outcomes of the remaining 118 attempts. Through its broad coverage, documented reproduction evidence, and common structure, AIFaultBench supports empirical studies of AI faults, debugging tool evaluation, practical failure investigation, and teaching with real-world examples.