Agent instruction files vary widely in adoption consistency and upkeep
Toward an Evidence-Based Template for Agent Instruction Files: An Empirical Study of Adoption, Consistency, and Maintenance
Software Engineering
Summary
Many AI coding assistants use special instruction files to learn how projects work, but there isn’t much data on how these files are created, whether they agree with each other, or how often they get updated. The authors looked at over 3,300 instruction files from 1,000 projects and found that many projects use multiple file types that sometimes give different advice. They also noticed that many files are hardly updated after creation. Based on what they found, the authors created a template to help write better instruction files and tested ways to automatically generate such files.
What this means in practice
- •For software development teams: Use the evidence-based template to create consistent and maintainable instruction files guiding AI coding assistants in projects.
- •For tool developers for ai coding assistants: Incorporate AGIT-Instruct benchmark to evaluate and improve automatic generation of agent instruction files for better AI guidance.
Authors
Mehedi Hasan Shanto, Md. Anaytul Islam, Muhammad Asaduzzaman, Alioune Ngom
Abstract
AI coding agents rely on repository-level instruction files to understand project conventions, development workflows, and constraints. However, different agents support different instruction formats, and limited empirical evidence exists about how these files are adopted, what information they contain, whether coexisting formats provide consistent guidance, and how they are maintained. We conduct an empirical study of 3,387 instruction files from 1,000 GitHub repositories across four formats: AGENTS.md, CLAUDE.md, GitHub Copilot instructions, and Cursor rules. We find that 42.2% of instruction-bearing repositories in the higher-popularity cohort maintain multiple formats. Among 386 manually examined pairs of corresponding sections, 16.1% provide divergent guidance. Our maintenance analysis further shows that 40.4% of the analyzed files have only one recorded modification event and that maintenance patterns differ across formats. Based on category prevalence and category addition ratios (CAR), we derive an evidence-based template containing ten Required, four Gap, and two Recommended categories. In an exploratory generation experiment, explicit template guidance increases structural category coverage from 18.7% to 82.8%. We further introduce AGIT-Instruct, a benchmark for evaluating instruction-file generation through structural coverage and content alignment. Across four generation configurations, retrieval-augmented generation achieves the highest full-file ROUGE-1 F1 of 0.459. A manual assessment of 278 generated category instances finds that 19.4% fully preserve and 61.2% partially preserve developer-written intent. Our findings provide empirical guidance for organizing and maintaining agent instruction