Summary
Assessing the security risks of artificial intelligence (AI) systems is tricky because current methods are often inconsistent and hard to verify. The authors created a new framework that uses clear, rule-based steps to evaluate AI security in a way that anyone can repeat and audit. Their approach links real evidence from AI projects to security techniques, scoring risks on a simple four-level scale and checking that the scoring rules are consistent and logical. They tested the framework on several open source AI projects and showed it could reliably detect security improvements and remaining risks after changes. This makes it easier to understand, trust, and repeat AI security assessments.
What this means in practice
- •For ai security teams: Use the framework to run consistent, repeatable security risk assessments across AI projects and verify improvements after hardening controls.
- •For devops engineers: Integrate security risk checks into continuous integration pipelines to automatically flag AI security concerns based on source code evidence.
Abstract
Artificial intelligence systems are increasingly deployed in high impact and safety critical settings, yet security assessment remains difficult to reproduce and defend under audit. Existing approaches often rely on narrative checklists or assessor driven scoring, and they lack an explicit, machine evaluable mapping from observable engineering artefacts to stable technique level outcomes. We present an evidence driven AI security assessment framework that operationalises assessment as a deterministic decision function. The framework normalises heterogeneous artefacts into a project independent Control ID taxonomy scored on a bounded four level ordinal scale, compiles technique level predicates from a pinned MITRE ATLAS snapshot via an explicit mitigation to control mapping, and outputs technique indexed feasibility and impact levels with traceable links back to the triggering evidence. We package all normative choices as a versioned assessment policy object to support repeatable reassessment across snapshots. To ensure semantic correctness, we formally verify boundedness, totality, ordered semantic consistency, and monotonicity of the compiled evaluator over the full declared score domain. We evaluate the framework on five public open source AI projects pinned to explicit repository snapshots, quantify before and after changes under a unified hardening intervention, and validate responsiveness to real engineering changes through fork based implementations of Software Bill of Materials (SBOM) generation and Continuous integration (CI) security scanning gates. Results show consistent downward shifts in feasibility profiles under strengthened observable controls, while worst case residual feasibility persists when technique specific core controls remain absent from the evidence scope.