Ai coding agent rules stay useful despite some recurring errors
Lessons Outlive Their Failures: A Case Study of Rules for AI Coding Agents
Software Engineering
Summary
When AI coding helpers make mistakes, people set up rules to catch and fix these errors. The authors studied one AI system over six weeks and found that most mistakes came from outdated explanations. They also showed that rules created for past problems often keep helping even after the original issue is fixed. However, some problems, like outdated documentation, keep coming back and still need checking. This means AI coding helpers need regular rule updates and checks to keep working well.
What this means in practice
- •For software development teams: Maintain AI-assisted codebases by regularly auditing and updating rules to catch recurring issues like stale documentation.
- •For devops engineers: Enhance continuous integration pipelines with automated rule-based checks to prevent previously fixed AI agent errors from recurring.
Tested on one dataset.
Authors
Hassan Hassanzadeh Aliabadi
Abstract
AI coding agents increasingly work inside a harness: rule files such as AGENTS.md, on-demand skills, runtime hooks and automated checks. Most studies of harnesses look at a single point in time or at short benchmark runs. I report a six-week case study of my personal system of record, a plain-markdown repository that three AI agent tools edited daily under one shared rule set. I built a ledger of 118 agent failures, their fixes and whether they came back. I also tested the system's monthly LLM documentation audit against 47 known stale statements and 20 planted errors. Finally, I replayed five past failures in isolated git clones, with and without their fixes. All results come from one repository with three runs per condition, so I report them as exploratory. Stale documentation (drift) was the largest failure class (48 of 118), although 35 of those were found on one audit day. A single audit run found about half of the known drift (Claude Sonnet 5.5 mean: 50%), and at least 93% of its findings were correct. On Sonnet 5.5, the agent finished four of the five replayed tasks. In three of those four, the failure did not happen, with or without its fix. Only the stale-count failure came back without its check, and the check stopped it in every run. On older models, an agent drafted an em-dash once while the em-dash lint check was active, and the check rejected it. A written rule on run-time estimates did not change Sonnet 4.5's overestimates. These results suggest that harness lessons can outlive the failures they were written to fix, while drift persists and still needs mechanical checks. Tools that evolve a harness add lessons faster than they retire them, so they should re-test their rules from time to time and retire the obsolete ones.