AI agents struggle to share info in team decisions but improve with training
TeamGym: A Gym for Team Decision-Making by AI Agents with Distributed Information
Artificial Intelligence
Summary
Making decisions as a team is hard when important facts are spread across different group members instead of all being known by one person. The authors created TeamGym, a test environment using real-world documents where AI agents must work together and share key facts to decide correctly. They found that current AI agents often fail to communicate and identify crucial information, lowering their chances of success. By training agents with TeamGym using reinforcement learning, the agents learned to communicate better and make better group decisions.
What this means in practice
- •For legal professionals: Improve AI tools that aid teams reviewing complex documents by better sharing and integrating crucial facts for group decisions.
- •For software development teams: Develop AI assistants that enhance team coordination by identifying and communicating key technical information across distributed developers.
Authors
Chanwoo Park, Jaywoong Jeong, Zachary Wojtowicz, Yubin Kim, Taekyun Lee, Kaiqing Zhang, Asuman Ozdaglar
Abstract
Many real-world decisions require teams to integrate information distributed across participants. Yet AI agents are still primarily trained and benchmarked in isolation. We introduce TeamGym, an environment for evaluating and training language model-based agents on team decision-making problems constructed from real-world legal, finance, medical, and software engineering corpora. TeamGym introduces a method to identify a minimal set of facts, each element of which is necessary and all of which are collectively sufficient for a correct final decision, which we call pivotal information. This enables TeamGym to both control the distribution of pivotal information and track how well agents identify and integrate pivotal information when making team decisions. Examining 8 language models on 12 source corpora, we find that current AI agents often fail to effectively share information: Distributing pivotal information across participants lowers success rate by 22.9 percentage points on average relative to giving the same information to a single model (a communication gap). We also document that participants do not effectively retrieve decision-relevant information on their own: A team's success rate is lowered by a further 19.9 points when participants must identify and extract pivotal information from source material rather than receiving it directly (a retrieval gap). To help remedy these deficits, we use TeamGym to train policies through reinforcement learning on team decision loss. Across two policy-training experiments, we find that training improves final-decision quality and induces emergent coordination in participation, speaker selection, and communication, even though the reward evaluates only the final decision.