Dataset reveals security flaws in AI generated C and C++ code
LLMVul: A Vulnerability-Labeled Dataset of LLM-Generated C/C++ Functions from Real Production Repositories
Software Engineering
Summary
Software developers increasingly use AI tools to write code, but the security risks of AI-generated code are not well understood. The authors created LLMVul, a large collection of real-world C and C++ functions written by AI from public software projects, labeled with known security problems. This dataset helps to identify common mistakes AI makes and supports better detection of vulnerabilities in AI-assisted coding. They verified the labels by combining automatic analysis with human review to ensure accuracy. LLMVul is publicly available to support safer AI-powered software development.
What this means in practice
- •For software security teams: Evaluate and improve tools that detect vulnerabilities specifically in AI-generated C/C++ code using a real-world labeled dataset.
- •For software developers using ai tools: Analyze common vulnerability patterns in AI-assisted coding to guide safer code generation practices in C and C++ projects.
Authors
Mohammad Farhad, Shuvalaxmi Dass
Abstract
Large language models (LLMs) are increasingly used to generate and assist with software development, yet existing vulnerability datasets largely focus on human-written code or controlled prompting environments. This limits the ability to study security weaknesses in LLM-generated code as it appears in real-world software projects. We present LLMVul, a vulnerability-labeled dataset of LLM-generated C/C++ functions mined from real production repositories. We mine AI-assisted development activity from GitHub over a 4 year period, from November 13, 2022 to September 3, 2026, using provenance signals such as commit metadata and AI-related authorship evidence. After filtering and deduplication, LLMVul contains 21,430 unique C/C++ functions from 226 repositories, together with repository, commit, function, provenance, and AI-tool metadata. We establish vulnerability labels using an ensemble of complementary static-analysis and pattern-based techniques and assign Common Weakness Enumeration (CWE) categories to confirmed vulnerable functions. To assess labeling reliability, we additionally conduct independent manual annotation and measure inter-rater agreement using Cohen's kappa ($k=0.79$). LLMVul contains 1,540 ensemble-vulnerable functions spanning 17 unique CWE categories, providing substantially more real-world LLM-generated vulnerable C/C++ functions than existing vulnerability-oriented LLM code benchmarks. By preserving both code-level vulnerability labels and generation/provenance metadata, LLMVul enables reproducible research on vulnerability detection, security evaluation of LLM-generated code, and analysis of vulnerability patterns in AI-assisted software development. The LLMVul dataset is publicly available at https://doi.org/10.5281/zenodo.22668216.