ProEvent: An Event-centric Benchmark for Proactive Agents
2026-07-20 • Artificial Intelligence
Artificial Intelligence
AI summaryⓘ
The authors created ProEvent, a test to see how well AI agents can keep track of a user's upcoming events from chat messages and help proactively. They found that current AI models, including advanced ones like GPT-5.1, often make mistakes such as reacting too much or missing when events are canceled. The work shows that these models have trouble understanding subtle hints and thinking from the user's point of view. This benchmark helps highlight where AI agents need to improve to assist users better without explicit instructions.
proactive agentsevent-centric assistancelarge language modelsuser timetable trackinginstant messagingevent cancellationresponse timingfirst-person perspectivebenchmark evaluation
Authors
Guanzhen Li, Liangming Pan, Leye Wang
Abstract
Proactive agents are expected to anticipate user needs and provide autonomous assistance by perceiving environmental context without explicit instructions. A fundamental capability of such agents is to identify and track users' upcoming events, enabling continuous and event-specific assistance. For example, by recording the time and location of a planned hike, an agent can deliver weather reminders in advance or provide navigation support before departure. However, existing works on proactive agents largely overlook event-centric assistance, and the open-ended nature of proactive assistance poses challenges for reliable evaluation. To bridge these gaps, we introduce ProEvent, the first event-centric benchmark designed to assess an agent's ability to proactively maintain a user's timetable based on ongoing instant messaging chats. ProEvent provides synthesized yet realistic chats that consider the dynamic interaction among users, concurrent chat threads, and noise in the real world, and evaluates proactive agents on response timing, single-step response correctness, and multi-step response correctness. Experiments on eight LLMs and pipelines reveal that current agents frequently overact and struggle with event cancellation. Notably, even GPT-5.1 only reacts correctly in 26.7% of scenarios. Further qualitative analysis reveals fundamental limitations of current LLMs as proactive agents, particularly in detecting implicit events and reasoning from the user's first-person perspective.