Security challenges when human and agent actions share software control
Security of Agent-Integrated Software: When Human Operations and Agent Actions Coexist
Cryptography and SecuritySoftware Engineering
Summary
Software that combines human users and intelligent agents working on the same tasks faces unique security challenges. The authors explain that protecting the software by securing the human side and the agent side separately is not enough. They organize the main security problems into four categories and show how current methods only partially solve them. Their work points to important future directions for making these mixed human-agent systems safer overall.
What this means in practice
- •For enterprise software architects: Ensure security policies remain effective when designing software that supports both human operations and automated agent actions on the same workflow.
- •For cloud service operators: Detect and manage complex security risks in systems where automated agents and users interact to maintain shared software states.
A position paper. It proposes an approach and reports no results.
Authors
Ding Yang, Yuchen Ling, Shengcheng Yu, Zhenyu Chen, Chunrong Fang
Abstract
Agent-Integrated Software (AIS) embeds an intelligent agent in a conventional application, supporting both human operations and agent actions. Human operations let users make precise changes and inspect results, while agent actions carry out routine or multi-step tasks. These complementary roles make coexistence a likely long-term feature of many software systems. Human operations and agent actions affect the same software state and can use one another's results. Therefore, security policies must remain effective across both paths. We argue that AIS security must be assessed at the level of the whole software system. Protecting the agent and the conventional software core separately does not establish that they are secure together. To guide security analysis of AIS as a whole, we organize the problems arising from this coexistence into four categories: context misuse, authorization violation, execution control, and effect integrity. Using these categories, we examine how current practices address the security problems in AIS and where their protection remains limited. Building on this analysis, we identify research opportunities in preserving information provenance, enforcing policy across operation paths, maintaining valid authorization over time, and managing persistent effects and recovery. This resulting perspective provides a conceptual framework for understanding and improving the security of AIS.