Predictive guardrails in AI agents do not always improve safety
Predictive Safety Guardrails Do Not Always Make LLM Agents Safer
Artificial Intelligence
Summary
Large language model agents sometimes need to predict future risks to act safely, but this study finds that these predictions are often unreliable or not properly used. The authors show that making agents rely more on predictions can slow them down and reduce their success without always preventing safety problems. Even when prediction quality improves or is verified locally, safety issues can still occur. This means that simply having better predictions is not enough if those predictions are not accurately and effectively guiding decisions.
What this means in practice
- •For ai system developers: Design safer AI agents by focusing on reliable and faithfully used predictions rather than just increasing prediction involvement.
- •For autonomous system engineers: Improve safety protocols in autonomous systems by critically evaluating how future risk predictions are integrated into control decisions.
Authors
Weixiang Zhao, Zhihao Liu, Guancheng Li, Yanyan Zhao, Bing Qin, Ting Liu
Abstract
Large language model agents increasingly take consequential actions in digital and physical environments, where safety risks may emerge only after multiple interaction steps. This has motivated world-model-driven predictive safety guardrails, which anticipate future consequences before execution. However, existing evaluations largely focus on end-to-end safety outcomes, leaving unclear whether safety gains arise from reliable prediction and faithful use of predicted risks. In this work, we systematically diagnose the prediction-to-action pipeline along three dimensions: how world-model predictions are integrated into decision making, whether the predictions themselves are reliable, and whether their content is faithfully used. Across seven policy models and two safety environments, we find that stronger prediction involvement does not yield monotonic safety gains and can reduce task completion while substantially increasing inference cost. We further find that world-model predictions remain insufficiently reliable, especially over longer horizons, where safety prediction degrades substantially relative to near-term forecasting. Finally, we find that safety gains can persist even under corrupted predictions, while locally verified predictions still leave subsequent safety violations, showing that predictive content is not faithfully incorporated into the decision process. Together, these findings expose a fundamental prediction-to-action gap in current predictive guardrails: producing more or seemingly accurate foresight is insufficient unless that foresight is both reliable and faithfully translated into safer decisions.