Benchmark tests if AI can match expert software documentation standards
DoGBench: Can Agents Meet Expert Standards for User-Facing Documentation?
Software EngineeringArtificial Intelligence
Summary
Creating clear and accurate user manuals for software is tough and usually done by expert writers. This paper introduces DoGBENCH, a test to see if automated agents can update and create software documentation just as well as professionals. It uses real open-source projects and checks if an agent correctly decides when updates are needed and produces good-quality documents. The results show current agents still fall short, especially missing details or making errors when explaining software features.
What this means in practice
- •For software development teams: Automatically detect and update missing or outdated user documentation for open-source software projects.
- •For technical writers: Assist in reviewing and generating software documentation patches to speed up documentation maintenance workflows.
Authors
Frances Liu, Manny Silva, Paige Calvert, Ayu Adiati, Sarah Sanders
Abstract
We introduce DoGBENCH (Documentation Generation Benchmark), to our knowledge, the first benchmark for generating and maintaining real user-facing software documentation. It asks whether an agent can produce documentation that experienced technical writers would accept in review. The benchmark contains 292 items from open source projects, including Helm, PostHog, and Mautic. Each item gives the agent a pre-change repository and a trigger, such as a code pull request or a reported documentation gap. The agent must first decide whether the documentation needs an update. For items that need one, the agent must produce an acceptable patch in one attempt. For items that do not need updates, the agent must abstain. Task-specific rubrics, validated with project maintainers, score each patch on accuracy, completeness, reader guidance, placement, and repository conventions. The composite score combines patch quality with correct abstention, and a score of 100 means an agent meets every requirement for the task. Scores should not be interpreted as a percentage of an expert's capability. We evaluated seven agents. The highest-scoring agent reached 47.3 out of 100 on the 117-item held-out split. In a separate audit of 1,267 patches, the most common failure modes were task-completion gaps (45.5%), technical inaccuracies (36.6%), and incomplete conceptual or reference coverage (32.5%). Analysis of the corresponding trajectories identified three key patterns associated with these failures: (1) describing interfaces without examining how readers use them (36.0%), (2) missing decisive evidence and filling the gaps with plausible assumptions (33.1%), and (3) stopping after finding the first plausible documentation surface and leaving other affected pages stale (30.1%).