Large language models generate test mocks with mixed accuracy and executability
Executable but Unfaithful: Mock Object, Stub, and Assertion Generation by LLMs and Coding Agents
Software Engineering
Summary
Testing software often needs pretend parts, called mocks, to check if the code works right. The authors created a test set to see how well advanced AI tools like large language models and coding agents can make these pretend parts from real code. They found these tools can produce runnable mocks but often make mistakes that tests might not catch. The study helps developers understand these tools’ strengths and where errors happen in simulated testing parts.
What this means in practice
- •For software developers: Use AI tools to automatically create test mocks and stubs to speed up unit test development when source code is available but runtime executions are not.
- •For quality assurance teams: Assess and improve the accuracy of automatically generated mock assertions to better identify interaction faults during software testing.
Authors
Hao Ning, Jiahui Dong, Zhen Yang, Guanda Yan, Yichen Sun, Xuanyu Zhu, Zheng Shen, Yifei Pei, Jacky Keung, Jie M. Zhang
Abstract
Mocking is a foundational practice in unit testing, enabling developers to isolate the code under test by replacing real dependencies with simulated alternatives. Automating the generation of mock objects, stubs, and mock assertions has been a long-standing research goal. However, capture-and-replay approaches rely on recording the runtime behavior of actual dependencies. Evolutionary approaches such as StubCoder overcome this requirement but constrain generation to predefined symbol pools. Large language models (LLMs) and coding agents offer an alternative by generating mocking artifacts from source code without requiring recorded dependency executions or fixed symbol pools. Yet, no systematic study has evaluated their capabilities across all three artifact types. To address this gap, we construct MockBench, a benchmark comprising 165 mock artifact generation tasks from real-world Java and Python projects, with validated reference mock objects, stubs, and mock assertions. We evaluate six LLMs and three coding agents in terms of executability, behavior fidelity, and interaction-fault detection capability, complemented by comparisons with existing tools and sensitivity analysis. Experimental results show that LLMs and coding agents indeed have been the state-of-the-art techniques in mock-based testing. More specifically, compared with LLMs, coding agents substantially improve executability, while these gains do not consistently translate into higher behavior fidelity or better interaction-fault detection. Furthermore, we analyze 660 failures from LLMs and coding agents, and identify ten error types across the three artifact categories, revealing behavioral deviations that passing tests can conceal. Our findings highlight the challenges of mock artifact generation and provide practical guidance for both developers and researchers.