Summary
Measuring how well enterprise AI systems perform is tricky because many factors influence their abilities, not just the AI model itself. The authors point out that current benchmarks mostly focus on the model identifier, ignoring details like how the system is served and connected. They propose a new protocol called IB2 that tests the entire serving setup to give a more accurate picture of the system’s real-world capability. Their tests showed that different serving routes using the same AI weights can perform differently, revealing limits that advertised model IDs don’t show. This finding suggests a better way to judge AI system performance by looking at the full delivery method, not just the model version.
enterprise AI systemsbenchmarksmodel identifierserving routemeasurement protocolevaluation contractreliability scoringcapability bindingmultiplicity adjustmentperformance discrimination
Abstract
Enterprises deploy systems, not checkpoints. Usable capability depends jointly on weights, serving route, precision, output contract, and harness, yet all 18 audited benchmarks score advertised model identifiers. We treat this as measurement error and give a protocol that makes it reportable. It has three parts. A gold-blind capability-binding preflight verifies that a route can execute the evaluation contract before any task reaches it; a reliability-inclusive first-pass scoring rule keeps failure in the score while keeping unsupported capability out; and adjudication is structurally score-blind. We call the protocol IB2 and release its algorithms, classification tables, request contract, and manifest schemas. Its reference instantiation, 128 locked tasks and 987 assertions over document, spreadsheet, chart, tool and database work, stays sealed: the procedure is the artifact, not the corpus. Across eleven systems, four results. Capability availability is measurable: two complete single-route runs on identical weights later failed distinct predicates of the finalized binding gate, while a third passed that gate before a fresh run. The advertised identifier exposed neither limit. Discrimination is not uniform: four of seven suites saturate under a six-system band, with the spread almost entirely from governed database work and multi-tab joins, so we report interval-backed resolution groups, not ranks; two of the nominal five-label output's four cuts fail multiplicity adjustment. Serving-arm choice moved one declared revision and precision from 77.38 to 82.54, paired interval [0.11,10.60], though the arms differ in access mode, harness generation, and the serving tool-call parser, and harness generation is a property of our evaluator, not any endpoint. Excluding failed responses from denominators changes the point ordering, so reliability inclusion changes a conclusion, not its wording.