Reviewers demand stronger evidence for AI authored code changes
Plausible Is Not Enough: Trust and Evidence in AI-Authored Pull Request Review
Software Engineering
Summary
This paper looks at how software developers review code suggested by AI versus humans. It finds that developers trust AI code less and want more proof that it works well before accepting it. Instead of relying on the AI author’s past work, developers focus on understanding the changes, running tests themselves, and who wrote the AI prompt. The authors suggest tools should show where AI code comes from and clearly mark what is verified by humans.
What this means in practice
- •For software development teams: Improve code review tools by highlighting AI-generated code provenance and uncertainty to support reviewer trust decisions.
- •For software quality assurance teams: Design verification practices that prioritize test inspection and partial reimplementation to validate AI-generated code before approval.
Authors
Md Shamimur Rahman, Khairul Alam, Banani Roy, Chanchal K. Roy
Abstract
Code review is where AI-authored pull requests (AIPRs) must earn reviewer trust before integration, yet it is unclear whether reviewers apply the same criteria used for human-authored pull requests (HPRs). We surveyed 239 practitioners from 31 countries using matched HPR and AIPR questions to compare trust, evidence requirements, and verification practices within respondents. Most respondents (64.9%) reported lower initial trust in AIPRs. They relied less on author track record and subsystem familiarity, emphasizing instead human prompter expertise and bounded change size. For approval, respondents more often considered their own reasoning through critical cases sufficient for AIPRs than HPRs (55.2% versus 28.5%), while relying less on test coverage. Overall, 61.1% reported higher evidence requirements for AIPRs, and the share who would block approval under residual uncertainty increased from 19.4% for HPRs to 43.9% for AIPRs. Respondents prioritized test inspection, project context, and local execution, and more often selected partial reimplementation to verify non-trivial logic. Unclear ownership was also more likely to pause AIPR review (29.3% versus 14.6%). Open-ended responses pointed to the difficulty of validating plausible code without reliable context and human accountability. These findings frame AIPR review as an evidence-ownership problem, not merely a correctness check. Reviewers need to connect generated changes to validated behavior and responsible maintainers. Review tools and agentic workflows should expose provenance, distinguish generated claims from human-validated evidence, and make uncertainty visible before approval.