ModelShield catches hidden malicious actions in shared ml models
I Don't Care How You Hide It, I Care What You Do: Runtime Detection of Malicious ML Models
Cryptography and Security
Summary
People can share machine learning models online that secretly do harmful things when used. The authors show that current detection tools miss many of these sneaky models because they look only at fixed code pieces and miss tricks like replacing banned functions with similar ones. They created ModelShield, a system that watches what the model actually does while running, tracking specific behaviors linked to the model’s actions to find malicious activity. Their tool caught all known tricky attacks and found harmful models on popular platforms without false alarms, and it runs fast enough to screen large collections of models.
What this means in practice
- •For machine learning platform operators: Scan and block malicious machine learning models on public AI model hubs to protect users from hidden attacks during model use.
- •For cybersecurity teams for cloud services: Detect and analyze runtime malicious behavior from machine learning models deployed in cloud environments to prevent exploitation.
Authors
Hala Ali, Andrew Case
Abstract
Pre-trained machine learning (ML) models shared through public platforms can be weaponized to execute malicious payloads during loading or inference. Existing scanners prove inadequate for detecting malicious models as they rely on static analysis and function denylists, support limited serialization formats, and miss attacks that replace denylisted functions with equivalent alternatives, exploit unsupported loading paths, or abuse legitimate framework APIs. To address this problem, we present ModelShield, the first runtime behavioral detection system for malicious ML models that attributes observed system-level behavior to specific model operations. It combines application-level instrumentation of ML framework APIs with eBPF-based syscall tracing, tracking model behavior across spawned threads and processes to separate model-triggered activity from environment noise. Attributed syscall sequences are classified using 195 detection patterns mapped to MITRE ATT&CK techniques. We implemented ModelShield to support 17 ML frameworks and 36 formats. On an independent dataset, it is the only evaluated tool to achieve zero false positives, with false negatives limited to payloads that produce no malicious runtime behavior. It detected all 29 published evasion variants spanning function substitution, loading-path obfuscation, and TensorFlow API abuse, where no existing scanner achieved complete coverage. Across 145,000 models scanned from public platforms, it captured 41 malicious models on Hugging Face, including a reverse-shell payload and a payload that performed host reconnaissance and outbound exfiltration. Among these, ModelScan, PickleScan, Fickling, and ModelAudit detected only 29%, 56%, 80%, and 93%, respectively. Despite runtime overhead, ModelShield has a median analysis time of 12.3 seconds, with 90% of scans completing within 35.5 seconds, making it practical for hub-scale screening.