AI agents struggle to time helpful chat contributions accurately
When Should AI Chime In? Benchmarking Timely Contribution of Proactive Agents in Multi-Party Chat
Artificial Intelligence
Summary
AI agents can help group chats by sharing useful information without being asked, but if they speak up at the wrong time, it can disturb the conversation. The authors created PITCH, a dataset where people judge the best timing for AI contributions in real group chats. They found that AI models often interrupt too early compared to humans, who tend to wait for better moments. This work highlights that AI needs to learn not only what to say but also when to say it in conversations.
What this means in practice
- •For chatbot developers: Improve timing algorithms so AI chatbots contribute relevant info without disrupting group talks.
- •For virtual meeting platform teams: Enhance group collaboration tools by integrating AI assistants that know when to intervene appropriately.
Authors
Ruotong Wang, Brian S. Yu, Xinyi Zhou, Joseph Chee Chang, Pao Siangliulue, Jonathan Bragg, Liwei Jiang, Amy X Zhang
Abstract
Proactive agents can support group collaboration by surfacing useful information without waiting to be asked, but even relevant contributions can be disruptive if poorly timed. In this work, we isolate the timing question: given a fixed candidate contribution such as a clarification or relevant fact, when should an AI agent chime in? We introduce PITCH (Proactive Intervention Timing in CHat), to our knowledge the first benchmark with large-scale human judgments of proactive agent timing collected from participants in the same unscripted multi-party conversations they took part in. To collect high-quality judgments, we design a novel interface in which 2-4 people chat synchronously, author candidate AI contribution messages, place their own and others' contributions into the chat history, and then deliberate as a group about the best placement for each message. PITCH contains 384 candidate contributions and 1,091 judgments, each marking an ideal turn, an acceptable window, or that a contribution should not appear, from 89 participants engaged in 35 conversations. Evaluating 15 models across 28 configurations, we find that while models can place contributions well when given the entire conversation history, they systematically chime in too early when asked to decide live as the conversation unfolds: best-turn accuracy falls from 57.8% to 19.0%, with nearly half of placements occurring before every participant's acceptable window. In comparison, lay humans deciding live are more accurate (23.7% vs 19.0%), and fewer of their placements are premature (39.7% vs 46.4%). Model reasoning traces show that 93% of intervention justifications cite relevance, while almost none consider waiting. By providing a human-grounded benchmark, our work reveals gaps in agents' ability to integrate into group settings and points towards proactive agents that can reason explicitly about appropriate timing.