Cortex improves coding agent skills by testing and pruning rules
Cortex: Measurement-Gated Evolution and Pruning of Coding-Agent Skills and Rules
Software EngineeringArtificial Intelligence
Summary
Coding agents use files with rules and skills written by developers to help complete tasks, but these can sometimes be incorrect or inconsistent. The authors created Cortex, a tool that checks fixes made by developers by testing whether each change actually helps the agent perform better across many tasks. Cortex keeps only changes that consistently improve results without causing new problems, and removes unused or harmful rules and skills. Their tests show Cortex helps coding agents follow project-specific rules more reliably and avoids keeping unhelpful or damaging instructions.
What this means in practice
- •For software developers: Automatically refine coding agent instructions to maintain helpful rules and remove ineffective ones during project development.
- •For automated code review teams: Improve consistency in automated coding assistants by ensuring only verified helpful skills and rules are applied to code tasks.
Authors
Samuele Giampieri
Abstract
Coding agents such as Claude Code also follow text files that developers write for them: a project file, skills the agent may choose to open, and rules that load automatically when the agent reads matching files. We call these files the agent's harness. Developers add to it whenever they correct the agent, but one success proves little: the same agent can pass a task once and fail it the next time. We present Cortex, a tool that tests changes to the harness instead of trusting them. It saves fixes from the developer's sessions as replayable tasks, finds the corrections that recur, writes one skill or rule at a time, and keeps it only if repeated runs with and without it show a net gain and no task made noticeably worse. Existing text is tested the same way before it is deleted, and Cortex neither adds nor deletes text the agent never used. We tested Cortex with Claude Haiku 4.5 on a repository built for the purpose, with conventions the model tends to break, one area with nothing to learn, and a script playing the developer. On held-out tasks, the harnesses it built made the agent follow the conventions far more often than no skills or rules did, left the comparison area unchanged, and added no text read on every turn. Rules, which load automatically, did most of the work; skills helped only when offered with the files a task touches, and those offered on every turn were rarely opened. Cortex kept none of the useless or harmful suggestions we planted, but the agent never opened the useless ones or the harmful skills, and it reached no verdict on the harmful rules, so the test shows that Cortex keeps no unused text, not that it detects damage. Under the stronger Claude Sonnet 5 a single run's harness added little, and on a real open-source project, where the agent was rarely corrected, Cortex found nothing to learn. We release the tool, the lab and its data.