The Patchwork Problem in LLM-Generated Code

2026-07-09Software Engineering

Software EngineeringArtificial Intelligence
AI summary

The authors explain that code generated by large language models (LLMs) often seems fine because it compiles and passes tests, but still breaks when actually used. This happens due to 'structural' issues, like missing configuration or inconsistent imports, which normal testing doesn't catch. They call this the "patchwork problem" and analyze it by representing code dependencies and relationships as graphs. The authors propose a new verification method combining existing static analysis with specialized detectors to find these hidden problems. Their tests show many structural errors are missed by usual tools and are common in real AI-generated code projects.

LLM-generated codestructural coherencestatic analysisconfiguration keysimport dependencycontrol-flow graphrouting graphsoftware verificationsoftware testingpatchwork problem
Authors
Viraaji Mothukuri, Reza M. Parizi
Abstract
LLM-generated code often compiles, passes tests, and appears correct, yet breaks once deployed. The root cause is frequently structural rather than logical. A generated endpoint references configuration keys never declared in the project, an import targets a package that does not exist in any registry, or a new route omits the authentication guard applied to every sibling endpoint. Each patch is locally valid but globally incoherent, and standard CI toolchains rarely surface these failures. As LLM-powered coding tools see widespread adoption, this blind spot poses a growing risk to software quality. We call this the \textbf{patchwork problem}. This paper formalizes structural coherence as consistency invariants over graph representations of repository artifacts, including import, call, dependency, configuration, schema, resource, control-flow, and routing graphs, and introduces an eight-category failure taxonomy distinguishing defects specific to LLM generation from those merely amplified by it. We present a hybrid verification framework that delegates to mature static analysis tools where they already excel and deploys purpose-built detectors for cross-cutting invariants underserved by existing toolchains, targeting provable constraint violations rather than heuristic pattern matching. Empirical evaluation across two frontier models under four prompting strategies reveals that the vast majority of structural failures evade type checking, testing, and SAST entirely, and that failure patterns diverge qualitatively between models in ways that challenge model-agnostic mitigation strategies. External validation on real-world AI-generated repositories confirms that these failures are not artifacts of controlled experimentation but are prevalent wherever LLMs write code with minimal human oversight.