Quality assurance in AI agents shows gaps in testing and safeguards
A Large-Scale Empirical Study of Quality Assurance Practices and Gaps in AI Agents
Software Engineering
Summary
AI programs that use large language models are getting more complex and can do many things automatically, but this also leads to new risks and mistakes. The authors studied 157 popular open-source AI agent projects to see how people test and protect these systems. They found that most testing focuses on basic features and some risky actions, but many important checks are missing. The projects often do not test how agents behave in tricky or multi-step situations, and safety measures are inconsistently applied. This means more thorough end-to-end testing is needed to keep AI agents working safely and reliably.
What this means in practice
- •For software engineering teams: Improve AI agent reliability by identifying overlooked test cases and applying consistent safeguards across execution paths.
- •For web automation developers: Develop more robust automated workflows by adding end-to-end checks for multi-step tool interactions and adversarial inputs in AI agents.
A survey. It maps existing work.
Authors
Wuyang Dai, Moses Openja, Jiho Shin, Hung Viet Pham, Song Wang
Abstract
Large language model (LLM)-based agents are increasingly used across software engineering, web automation, research, and productivity applications. Their integration of planning, memory, tool use, code execution, and external interactions enables greater autonomy but also introduces new reliability, safety, and security risks. We present a large-scale empirical study of quality assurance (QA) practices in 157 open-source LLM-based agent projects with at least 100 GitHub stars. We analyze documentation, source code, configurations, and tests to characterize QA practices across execution surfaces, safeguards, testing artifacts, risk scenarios, and recurring gaps. We find that current QA primarily focuses on basic functionality and high-risk actions, while coverage remains fragmented. Safeguards are inconsistently applied across equivalent execution routes, tests rarely examine boundary, adversarial, or multi-step tool-use failures, and identified risks are seldom translated into end-to-end QA checks. These findings highlight the need to move beyond feature-level testing toward systematic end-to-end validation that ensures agent workflows remain within intended boundaries when interacting with untrusted inputs, tools, persistent state, and external APIs.