Large language models assist industrial test failure root cause analysis
Supporting Industrial Test-Failure Analysis with LLM-Based Systems: An Experience Report
Software Engineering
Summary
Companies run lots of automatic tests each night to find problems in their software, but figuring out why tests fail can be slow and hard because the information is scattered in different files. The authors tried using smart AI helpers, called Large Language Models, to automatically analyze these test failures by reading test details and logs. They tested two setups: one where a single AI agent worked alone, and another where multiple AI agents worked together. Both helped, but the simpler single-agent system was faster, cheaper, and just as good at explaining test failures in the cases they studied.
What this means in practice
- •For software testing teams: Use single-agent AI systems to quickly analyze nightly test failures and generate understandable root cause reports across test logs and metadata.
- •For software quality assurance teams: Apply AI-driven root cause analysis workflows to reduce manual inspection time for diverse test data and logs in industrial environments.
Authors
Eric Jansson, Per Strandberg, Thomas Sörensen, Eduard Paul Enoiu, Wasif Afzal
Abstract
This study examines tool-augmented Large Language Model (LLM) systems for supporting Root Cause Analysis (RCA) of nightly test failures at Westermo Network Technologies AB. Nightly test executions produce heterogeneous test data and logs that practitioners currently inspect manually across multiple sources. We implemented an RCA workflow in single-agent and orchestrated multi-agent configurations, both with access to test metadata and logs. An exploratory industrial case study used two real failure scenarios. Six practitioners evaluated the scenario reports through a survey and focus group, and operational measurements were collected from 120 repeated executions. The evaluation covered practitioner-perceived correctness, reasoning quality, fix realism, clarity, usefulness, and trust, as well as cost, duration, and consistency. Neither configuration showed a consistent practitioner-perceived quality advantage across the two scenarios. The single agent system generated reports faster and at lower cost, making it the more practical baseline in this context. The potential benefits of agent architectures require further evaluation in more complex scenarios.