AI agents use risk scores to control risky actions during tasks
RATION: Towards Quantitative Risk-Based Access Control for AI Agents
Cryptography and SecurityArtificial IntelligenceMachine Learning
Summary
AI agents sometimes do unexpected or risky things when trying to help with complex tasks. The authors created a system called RATION that watches how risky the agent's actions are as it works and sets rules based on those risks. This system mixes rules from the user's request and lessons from past similar tasks to keep actions safe without stopping the agent from finishing the job. The authors tested RATION on several security checks and found it keeps agents safe while still letting them do useful work.
What this means in practice
- •For ai system developers: Manage risks in AI agents by tracking and controlling risky actions during task execution to improve security without reducing usefulness.
- •For enterprise security teams: Integrate risk-based access controls to prevent security breaches from AI agents while maintaining operational productivity.
Authors
Yuntao Du, Harsh Chaudhari, Yuetian Chen, Ron Heichman, Ashish Jayamohan, Kumar Pallav, Mahim Patel, Kaiyuan Zhang, Kyle Polley, Jerry Ma, Ninghui Li
Abstract
AI agents increasingly interact with external environments to complete complex user tasks. These interactions can lead to unexpected behavior and expose agents to security threats such as indirect prompt injection. Current agent security systems largely protect agents by restricting their actions, which can result in a substantial security-utility trade-off for complex, open-ended tasks. To address this, we propose a quantitative risk-based approach to access control for AI agents that regulates measurable risks accumulated during task execution. As a concrete realization of this approach, we introduce RATION, a security framework that tracks measurable risks associated with agent actions and enforces quantitative constraints throughout execution. RATION first constructs a task-specific policy by combining explicit constraints derived from the user request with implicit risks calibrated from historical executions of similar tasks. During task execution, RATION continuously tracks the accumulated risks and applies different enforcement strategies depending on the type of risk. Extensive experiments on four security benchmarks show that RATION provides strong security while maintaining high task utility across diverse models and environments.