On Automated and Explainable Provenance of AI-Generated Code
2026-08-03 • Software Engineering
Software Engineering
AI summaryⓘ
The authors explain that while AI tools that generate computer code are very helpful, it's hard to understand where the generated code exactly comes from. Current tools only show problems after the code is made but don't explain why the AI produced that code or how to fix it in the future. The authors argue for new tools that can trace the origin of generated code back to the input prompts, training data, and model parts involved. They back this up with studies showing that this kind of tracing is needed for developers and legal experts. Their work outlines the challenges and research needed to build these explainable tracing systems.
Generative AICode generationExplainable provenanceTraceabilityTraining dataPost-hoc analysisModel interpretabilitySoftware developmentComplianceCausal analysis
Authors
Alejandro Velasco, Nathan Wintersgill, Trevor Stalnaker, Oscar Chaparro, Denys Poshyvanyk
Abstract
Generative AI for code generation has transformed software development, but it has also introduced a critical transparency problem: the origins of AI-generated code are opaque to the developers who use it, the organizations that deploy it, and the compliance professionals responsible for ensuring its legal and quality standards. Existing mitigations flag problematic outputs after the fact without explaining why a model produced them or how future generation could be improved. We present a research vision, grounded in a U.S. NSF-funded research grant, that argues that the next generation of CodeGenAI tools must be built on a foundation of explainable provenance: automated, post-hoc traceability that links generated code back to the prompt components, training data instances, global data features, and internal model components that caused its generation. We grounded this vision in empirical evidence from studies of software developers, model users, and compliance/legal professionals, which show that provenance information is a practical necessity that current tools do not provide. We characterize the problem across four traceability dimensions, outline a research program combining large-scale empirical studies with post-hoc causal and interpretability techniques, and identify the key open challenges that the community must address to realize this vision.