LLM agents learn to improve safety and success during real tasks
SafeCoEvo: Co-Evolving Safety Harnesses and Guards for LLM Agents at Test-Time
Artificial Intelligence
Summary
Large language model agents often face new jobs and risks they haven't seen before, and they only find out if they did well after finishing those jobs. The authors developed SafeCoEvo, a system that helps these agents learn from their past experiences while working so they get safer and better over time. SafeCoEvo uses two ways to learn: one quickly updates safety rules from recent tasks, and the other slowly improves how the agent judges risks overall. Their approach lowered unsafe results and increased task success compared to previous methods.
What this means in practice
- •For ai system operators: Improve deployed AI agents’ safety and effectiveness by enabling continuous learning from past task outcomes during real-time use.
- •For software engineers: Develop adaptive safety mechanisms in language-based AI systems that update both short-term rules and long-term risk models automatically.
Authors
Yu Cheng, Yongkang Hu, Shuaijie Ma, Zhihang Lin, Weicheng Meng, Jingyang Qiao, Jiuan Zhou, Yushuo Zhang, Yihang Chen, Weilin Luo, Kun Shao, Dong Li, Zhizhong Zhang, Yuan Xie, Zhaoxia Yin
Abstract
LLM agents deployed in real-world environments continually encounter new tasks and safety risks, while execution feedback typically becomes available only after each task is completed. However, existing self-evolving approaches commonly rely on multiple rounds of optimization over fixed and repeatedly accessible task distributions, fundamentally differing from test-time adaptation in real-world deployment, where only experience accumulated from past tasks can be used to improve safety decisions on future unseen tasks. To address this limitation, we propose SafeCoEvo, a test-time Harness-Guard co-evolution framework for LLM agent safety that enables the external safety system to continually adapt from accumulated runtime experience. SafeCoEvo jointly improves two complementary safety capabilities at different timescales: S-Harness rapidly externalizes recent runtime experience into updatable explicit safety knowledge that can promptly influence subsequent tasks, while GuardVPO internalizes accumulated runtime safety experience over a longer timescale into parametric risk-judgment capabilities. By combining short-term rapid adaptation with long-term capability consolidation, SafeCoEvo continually improves the agent's safety capabilities, reducing the unsafe outcome rate by 10.05% while improving the task success rate by 12.15% over the strongest baseline, thereby achieving simultaneous gains in safety and task utility.