Multi agent large language models increase defects found but also false alarms
The More the Better? A Comparison of Single- and Multi-Agent LLM Setups on a Financial Review Task
Artificial IntelligenceComputational Engineering, Finance, and Science
Summary
This paper studies whether splitting a large language model into multiple smaller agents improves its ability to review financial credit memos. The authors compare one big model checking all tasks at once to a team of smaller agents each doing part of the task. They find the team catches slightly more defects but also raises more false alarms, and uses more computing power. Showing the smaller agents the tasks they were not assigned reduces false alarms. Trying multiple single-agent runs can outperform the team. Overall, using multiple agents has small trade-offs depending on instructions and model type.
What this means in practice
- •For financial compliance teams: Determine if splitting document review tasks across multiple AI agents improves defect detection while managing false alarms in credit memo assessments.
- •For legal document reviewers: Assess trade-offs between single and multi-agent AI setups to optimize accuracy and efficiency in checking compliance requirements.
Tested on simulated data.
Authors
Pei Liu
Abstract
Practitioner guidance holds that a large language model reviews a document better when the work is divided among agents. We test a narrow form of it in a controlled comparison of single-agent and multi-agent setups on a financial task, reviewing credit assessment memos against requirements. One agent checks six requirements in one call. A team of three on the same model checks the same memo, two requirements each, in the same words, with no roles or later stage. Labels are known for every requirement on 300 synthetic memos, so misses and false alarms are measured separately. Two primary predictions are tested on four models at a Bonferroni-corrected level. Under an instruction to decide each requirement once, the team catches 0.08 to 0.10 more of the defects that carry an explanation on the three models whose single agent misses some, at the corrected level on two. It raises more false alarms on all four, at the corrected level on three. Balanced accuracy, an exploratory measure, changes by 0.02 or less, and the team uses 2.2 to 2.4 times the tokens. Showing each agent the requirements it was not given reduces the added false alarms on the two Claude models, at the corrected level on one. In exploratory comparisons, sampling the single agent three times and flagging on any draw gives higher balanced accuracy than the team on those models, and the team's results depend on the instruction.