Enterprise AI system performance measured by serving route not model ID
IBIB: A Protocol for Measuring Enterprise AI Systems by Serving Route, Not Model Identifier
Computation and LanguageArtificial IntelligenceMachine Learning
Summary
Measuring AI in companies is tricky because AI systems aren't just a single model but a combination of software and how it is served or run. The paper shows that measuring only by the model's name or version misses important details, leading to errors in performance evaluation. The authors propose a new testing method called IB2 that checks AI systems based on the actual route used to serve the AI, including reliability and how well it meets task conditions. Their method reveals that even the same AI model can behave differently depending on how it is served, which is important for understanding real-world AI capabilities.
What this means in practice
- •For enterprise ai teams: Measure and compare AI system performance based on actual serving setups rather than model versions for more accurate capability reporting.
- •For ai deployment engineers: Use a standardized protocol to include response failures and serving route differences in AI system evaluations to improve reliability understanding.
Authors
Blake Stenstrom, Charangan Vasantharajan, Brian Sathianathan
Abstract
Enterprises deploy systems, not checkpoints. Usable capability depends jointly on weights, serving route, precision, output contract, and harness, yet all 18 audited benchmarks score advertised model identifiers. We treat this as measurement error and give a protocol that makes it reportable. It has three parts. A gold-blind capability-binding preflight verifies that a route can execute the evaluation contract before any task reaches it; a reliability-inclusive first-pass scoring rule keeps failure in the score while keeping unsupported capability out; and adjudication is structurally score-blind. We call the protocol IB2 and release its algorithms, classification tables, request contract, and manifest schemas. Its reference instantiation, 128 locked tasks and 987 assertions over document, spreadsheet, chart, tool and database work, stays sealed: the procedure is the artifact, not the corpus. Across eleven systems, four results. Capability availability is measurable: two complete single-route runs on identical weights later failed distinct predicates of the finalized binding gate, while a third passed that gate before a fresh run. The advertised identifier exposed neither limit. Discrimination is not uniform: four of seven suites saturate under a six-system band, with the spread almost entirely from governed database work and multi-tab joins, so we report interval-backed resolution groups, not ranks; two of the nominal five-label output's four cuts fail multiplicity adjustment. Serving-arm choice moved one declared revision and precision from 77.38 to 82.54, paired interval [0.11,10.60], though the arms differ in access mode, harness generation, and the serving tool-call parser, and harness generation is a property of our evaluator, not any endpoint. Excluding failed responses from denominators changes the point ordering, so reliability inclusion changes a conclusion, not its wording.