COVER: Identifiable Evaluation of Coalition Routing
Artificial Intelligence
Summary
The authors present a new evaluation method called COVER to better understand how team changes in multi-agent systems affect final outputs and accuracy. Their approach fixes certain conditions to isolate the impact of routing choices, enabling more precise measurement of performance differences (regret) between policies. They test this method on datasets like MuSiQue-12 and HotpotQA-4, showing improvements in some regret metrics but also revealing limits on claimed routing gains. COVER is intended as a careful, auditable testing framework rather than proof that one routing method is universally best. Overall, the authors highlight how their method helps diagnose and measure routing effects without overstating results.
Authors
Raghul Sugumar, Amrit Gopinath
Abstract
When a multi-agent system changes its team, it also changes the messages and final answer it produces, so an end-to-end accuracy gap does not by itself identify a routing effect. We introduce method, an evaluation contract that fixes a public information boundary, downstream stack G, and finite legal team family before outcomes are generated. Complete coverage identifies exact finite-benchmark oracle regret conditional on that stack. For any finite collection of frozen policies, executing the union of their distinct selected teams is the minimal assumption-free support for every pairwise policy contrast, though not for absolute oracle regret. Two controlled tables with source-ID-disjoint splits test the instrument. On MuSiQue-12, a pre-specified privileged positive control improves regret from 0.532 to 0.402; a later public-interface control reaches 0.424 versus 0.554 but is retrospective. On HotpotQA-4, a pre-specified public direct scorer improves regret from 0.313 to 0.110. In fixed-stack Llama execution, verified route regret improves by 0.190, while the raw-answer gain is 0.010 with an interval crossing zero. A five-family ToolSandbox variant-shift validation exhaustively evaluates 16 declared teams on 14 untouched task variants (224/224 valid rows): the declared-family oracle reaches 0.768 safe-evidence completion, while the prospectively frozen router gets 0.637 (regret 0.131), failing the predeclared 0.10 criterion. A later retrospective comparator reaches 0.655, matching all-workers with 4.57 versus 5.00 workers on average. Thus COVER exposes selection headroom without manufacturing a routing win. A crossed-stack diagnostic shows absolute scores depend on G but finds no detectable router-by-finalizer interaction. COVER is an auditable measurement methodology, not a claim of stack-invariant or universal agent-routing superiority.