LLM choice impacts Python test reliability more than prompt style
Evaluating the effectiveness of class-level LLM-generated test suites in Python
Software Engineering
Summary
Generating Python unit tests with language models often produces many tests, but having lots of test coverage does not guarantee the tests run well or catch bugs. The authors studied how different ways of prompting and which LLMs are used affect how reliably the tests run, how much code they cover, and how good they are at spotting faults. They found the model used matters more than the prompt style and that test coverage alone isn’t enough to judge quality. They recommend making sure tests can run before checking their quality and using mutation tests together with coverage metrics.
What this means in practice
- •For software development teams: Choose the best language model before tuning test generation prompts to create more reliable Python test suites automatically.
- •For quality assurance engineers: Use mutation testing combined with coverage and executability checks to better assess automatically generated unit tests' fault-detection ability.
- •For automation tool developers: Incorporate model selection heuristics with prompt adjustment to improve AI-driven test generation tools for Python codebases.$Commercial implications: Enables building commercial AI-powered testing software that reliably creates executable and effective test suites.
Authors
Bilal Al-Ahmad, M. Harshvardhan, Khaled El-Fakih, Anas AlSobeh
Abstract
Context: Large language models (LLMs) can generate unit tests quickly, but high structural coverage does not establish that those tests execute reliably or detect faults. Existing evidence often treats coverage as the principal outcome and rarely compares prompt strategies and models through mutation testing at class level. Objective: This study examines how prompt strategy and model choice shape the executability, structural coverage, fault-detection effectiveness, and structural quality of LLM-generated Python test suites relative to human-written suites. Method: We evaluate multiple prompt strategies across a diverse set of current LLM configurations on the ClassEval benchmark. The evaluation combines execution outcomes, line and branch coverage, Cosmic Ray mutation scores, and structural quality indicators. Primary analyses treat successful execution as a prerequisite; paired comparisons use only classes shared by the relevant executable subsets. Results: Structural coverage is consistently near its ceiling and offers little discrimination among configurations. Executability varies substantially. The proposed prompt performs strongly for mutation score, but no prompt dominates across models. Model choice explains more variation than prompt choice, and their interaction shows that prompt effectiveness depends on the selected model. Human and LLM suites are evaluated on unequal executable subsets, so their relative mutation scores do not establish superiority. Conclusion: Reliable assessment of LLM-generated tests should treat executability as a gate and combine coverage with mutation testing and structural quality indicators. In practice, model selection should precede prompt tuning.