Large language models risk expanding AWS permissions beyond trust limits
IAMRAGBENCH: Benchmarking Authorization Expansions in LLM-Generated AWS IAM Policies under RAG Poisoning
Cryptography and Security
Summary
AWS uses policies to control who can do what in the cloud. This paper finds that AI tools generating these policies can sometimes create permissions that give too much access, which could let attackers do harmful things. The authors built a test benchmark to see how often this happens and showed that AI-generated policies need careful checking before they are used. Their work warns cloud teams to treat AI-created policies as suggestions that must be verified independently.
What this means in practice
- •For cloud security teams: Detect and prevent unauthorized AWS permissions when using AI-generated IAM policies by applying the IAMRAGBENCH benchmark tests before deployment.
- •For cloud automation engineers: Improve safeguards in automated policy generation workflows to flag AI-produced policies that risk over-privileging cloud resources.
Authors
Hasan Kaplan, Indika Kumara, Willem-Jan van den Heuvel, Damian Andrew Tamburri
Abstract
Large language models (LLMs) using retrieval-augmented generation (RAG) can synthesize Amazon Web Services (AWS) Identity and Access Management (IAM) policies from tasks and operational documents. Attacker-controlled context may induce provider-valid policies that exceed trusted authorization requirements. We present IAMRAGBENCH, a 60-case benchmark spanning scope, conditions, explicit denies, trust, permissions boundaries, and role delegation. It combines deterministic authorization-envelope scoring with case-specific validation, including request-level checks where supported. An attack requires a provider-valid clean policy that satisfies the task and a provider-valid treatment policy that violates the envelope, matches a predeclared goal, preserves an intended request, and newly authorizes a forbidden request. Across five endpoints, included-context exposure produces 49 exact authorization deltas, including 46 goal-matched attacks (46/292; 15.75\%). End-to-end BM25 over 720 documents retrieves the designated poison for all 60 tasks at \(k=5\) but produces 7/294 goal-matched attacks (2.38\%). Presenting the same weakening as a final security architecture exception produces 74/116 attacks in a frozen two-endpoint stress test (63.79\%) and 115/294 in a post hoc all-five extension using the same frozen prompts (39.12\%). In a replay of 194 provider-valid treatment policies that violate task requirements, AWS \texttt{CheckNoNewAccess} reports no new access for four of 168 goal-matched exact attacks. The full pre-deployment rule rejects all 171 exact deltas and routes five unresolved cases to review. These findings support treating IAM copilots as untrusted proposal generators whose outputs require independent authorization checks before deployment.