Vulnerable ai models widely used across many open source projects
Vulnerable Models in the AI Supply Chain: An Empirical Study of Adoption, Propagation, and Usage Patterns
Cryptography and SecuritySoftware Engineering
Summary
There are many AI models available for developers to use, but some have serious security problems like hidden backdoors. The authors studied almost 17,000 of these unsafe models on a popular AI hub called HuggingFace and found that thousands are still used in over 5,000 open-source projects. They discovered that developers often do not check where the models come from or attempt to fix the security issues when adding them to their software. This shows that just identifying unsafe models is not enough; better tools and rules are needed when developers integrate AI models into applications.
What this means in practice
- •For software developers: Identify risks and improve security checks when integrating AI models into software projects by analyzing real usage patterns of vulnerable models.
- •For ai platform providers: Enhance model hosting services with context-aware security features to better detect and prevent unsafe AI models from propagating into downstream applications.$Commercial implications: Enables building commercial model registry platforms with integrated security guardrails to protect users from vulnerable AI models.
Authors
The Anh Nguyen, Triet Huynh Minh Le, M. Ali Babar
Abstract
The rapid integration of Artificial Intelligence (AI) models has established a complex supply chain, but many of these shared artifacts harbor severe vulnerabilities such as insecure serialization and architectural backdoors. While prior efforts have extensively scanned model registries to flag malicious artifacts, a critical gap remains in understanding how developers actually adopt and integrate these unsafe models into downstream software systems. We bridge this gap by conducting a large-scale empirical study on the usage and propagation of 16,977 known vulnerable models on HuggingFace flagged by InsightDB. We track their prevalence across GitHub repositories, software packages, fine-tuning chains, and deployed applications. Our analysis reveals that, despite being publicly flagged, these vulnerable models appear in over 21,000 code locations across more than 5,000 open-source repositories, with 3,757 distinct flagged models (22.1\%) actively referenced in at least one project. To understand how these models are integrated in code and identify recurring usage patterns, we categorize and analyze the recurring patterns surrounding them via a customized pipeline that combines Code Property Graphs extraction with a tailored Frequent Subgraph Mining approach. Across all patterns, we find no evidence of provenance verification or model sanitization logic, with developers frequently bypassing security parameters or delegating trust to permissive framework defaults. These findings underscore that scanning model files alone is insufficient, highlighting the urgent need for context-aware security analysis and stricter integration guardrails at the application layer of the AI supply chain.