Multi-agent screening shows limits of agreement and acceptance evidence

Multi-Agent Screening: What Agreement and Acceptance Cannot Tell You

Multiagent SystemsArtificial IntelligenceInformation Retrieval

Summary

When multiple AI agents are used to filter items for a human reviewer—like sorting documents or papers—only the accepted items get reviewed. The authors show that just because agents agree on an item, it doesn't mean they are correct. Also, seeing which items the human accepts tells us about accuracy but only partly about how many relevant items are found. To estimate how many relevant items might be missed, it’s necessary to check a random sample of the items rejected by the agents. This method works regardless of the number of agents involved.

What this means in practice

  • For legal document reviewers: Estimate the number of missed relevant documents by sampling rejected items in multi-agent e-discovery systems.
  • For peer review coordinators: Improve understanding of error rates in AI-assisted paper screening by analyzing samples of unaccepted submissions.

Authors

Benjamin Chang, Dell Zhang

Abstract

When AI agents screen items for a costly human assessor, as in agentic search, e-discovery and AI reviewing of papers, only what they pass gets judged. We show what each kind of evidence can establish about their errors, even with unlimited data. Agreement shows that agents concur, not that they are right, however many there are. Acceptance by the assessor gives precision but only a lower bound on recall. Knowing which agents found which relevant items gives an upper bound on recall if agents' finds are not negatively correlated, and caps what one more agent can add if agents are interchangeable. Without assumptions about the agents, recall can be pinned down only by judging items they did not pass: a random sample of these estimates how many relevant items were missed, to any chosen accuracy, and the sample needed does not grow with the number of agents.