AI agent risk assessment improves change approval using logs and reruns

agent-risk: Risk Assessment for Changes to AI Agents

Artificial IntelligenceComputation and LanguageMachine Learning

Summary

Changing an AI agent after deployment is risky and requires approval. The authors studied how to use recorded logs to replay the agent's decisions and assess risks without full knowledge of all internal changes. They found that logs alone can be insufficient to guarantee correct decisions, but combining logs with saved states and targeted replay of certain episodes can better predict the agent’s behavior. This helps organizations design better evidence and testing to safely approve AI agent updates.

What this means in practice

  • For ai operations teams: Use logged data and targeted reruns to better evaluate safety risks before approving AI agent updates in production.
  • For software testing teams: Design evidence and save states in advance to enable precise risk assessments and safety gating during AI software changes.

Authors

Yue Zhao, Zixiang Xu, Angela Huyue Zhang, S. Alex Yang, Xiyang Hu

Abstract

Before changing a deployed AI agent (a stricter safety gate, a new model), organizations must approve the change, starting from existing logs. Extending classical risk analysis to partial records and partial changes to running systems, agent-risk replays logged episodes, assuming nothing else changes and the agent does not anticipate the change. The log settles episodes acting as before, a declared stop value settles stopped ones, and the rest need logged support, a saved-state rerun, or a bound. We prove what logs must record for exact replay and which saved states serve a gate family. For independent episodes, we bound the reruns a decision needs and tolerable drift from live behavior. In a bash sandbox, a 7,200-episode log alone decided neither of two retry gates, which let the agent act after refusals the log never shows. Rerunning only each gate's blocked episodes from saved states reached both live decisions at half the agent calls of a live test sized post hoc for them, given the log. Without per-step harm, a stopping gate's certified harm ceiling rose up to 2.5-fold, since the log hides whether harm preceded the stop. Approval thus needs evidence designed in advance: log fields, saved states, targeted reruns.