Capable but Careless: Do Computer-Use Agents Follow Contextual Integrity?
2026-06-22 • Artificial Intelligence
Artificial IntelligenceComputation and Language
AI summaryⓘ
The authors study software agents that help users by working across different apps like email and calendars. They identify a privacy problem where these agents accidentally share private info from one app while working in another. To test this, they created AgentCIBench, a tool that checks for three common mistakes these agents make, like sharing info with the wrong people or mixing up tasks. They tested 15 agents and found many leaked private info often. The authors hope their tool will help make future agents safer before release.
computer-use agentsprivacy riskcross-application accessAgentCIBenchvisual co-locationtask-ambiguity oversharerecipient misalignmentprivacy leakagesoftware evaluationautomated agents
Authors
Anmol Goel, Iryna Gurevych
Abstract
Computer-use agents (CUAs) now act on a user's behalf across personal applications such as email, calendars, and to-do lists. This cross-application access is useful, but it also creates a privacy risk that has been largely overlooked: when an agent works in one context, it can pull in information from another that is inappropriate in that context. Hence, we introduce AgentCIBench, an evaluation harness that turns this risk into executable, deterministically scored scenarios. We target three common failure modes in CUAs: visual co-location, where the agent pulls in prohibited items that sit next to the task target in the UI; task-ambiguity overshare, where the agent dumps dense personal state in response to an under-specified prompt; and recipient misalignment, where the agent sends content to an addressee for whom it is inappropriate. We evaluate 15 frontier agents and find a surprisingly high failure rate: 11 of 15 leak on more than 50% of scenarios, with an average leakage of 67.9%, and the same failures persist when agents act end-to-end in the environment to complete the task. We release AgentCIBench to encourage the development of safer computer-use agents and position contextual disclosure testing as a pre-deployment safety check.