AI-assisted software development guided by misfit detection and human review
A Design Theory for AI-Assisted Software Development Derived from Christopher Alexander's Theory of Form
Software Engineering
Summary
Code created by AI language models might not perfectly match the requirements given by human developers. The authors propose a new method to check and fix AI-generated code by looking for where the code doesn't fit well with the project’s usual way of working. Their approach combines automated checks with human judgement in two loops: one where the AI tries to improve code with automatic gates, and another where humans decide if the code really fits the needs. They tested this on a software system with many automated tests and rules to catch problems, showing how this method can help manage AI-generated code.
What this means in practice
- •For software development teams: Improve reliability of AI-generated code by integrating automated misfit detectors with human review in a two-loop development process.
- •For quality assurance teams: Use explicit representations of local coding traditions and problem patterns to design tests and gates that catch AI code misfits before deployment.
Authors
Chien-Tsun Chen, Yu Chin Cheng
Abstract
Code generated by large language models (LLMs) cannot be assumed to meet specified requirements. Reviews, testing, and static analysis still apply, but which of them a sufficient harness needs, and in what role, is open. We propose a design theory derived from Christopher Alexander's theory of form, and a methodology for applying it. In Alexander's account, fit between a form and its context can be perceived only negatively, through the absence of identified misfits. We make the organization's tradition explicit and derive the misfits from it and from the problem's classification. The theory models the LLM as a non-native vernacular builder, trained on many codebases but native to none, whose output tends to drift toward mainstream conventions rather than the local tradition. We engineer four pieces of machinery: explicit representations of the problem (Jackson's problem frames) and of the tradition (a four-form pattern language); deterministic misfit detectors; a fix loop; and a human-gated legislative circuit governing the representations and detectors. We call the resulting methodology, a practice of harness engineering, Misfit-Governed Development (MGD). Its dual-loop process separates an autonomous inner loop, where the LLM iterates against the gates, from a human outer loop, where specifications are judged against the world. Together they form the S = P = T = W assurance model (specification, program, tests, world), whose equals signs name relations, not identity. We report evidence from building and rebuilding a Scrum system of four event-sourced aggregates from 64 problem-frame specifications, verified by about 1,300 generated tests and 28 blocking gates, one applying 188 rules. This addresses the generativity dimension of Alexander's 1996 OOPSLA challenge. The moral dimension, whether the specification still fits the world, requires human judgment and belongs to the outer loop.