Learned routing evaluation separates claim strength from subject of evidence

When Evidence Changes the Subject: Subject-Typed Claim Licensing for Learned Routing

Artificial Intelligence

Summary

When complex computer systems combine learned parts with other methods like search or repair, it's hard to tell which part actually caused their success. The authors show that judging just how strong evidence is isn't enough; you also need to know which component the evidence supports. They propose a method called subject-typed claim licensing to clearly separate evidence about one system part from evidence about the whole. Testing this approach in a routing task showed it helps reviewers make more accurate judgments about specific claims and credit different parts correctly. This means future evaluations should consider both how strong evidence is and which part it supports.

What this means in practice

  • For machine learning engineers: Evaluate individual learned components within complex AI systems more accurately by separating strength of evidence from the component it applies to.
  • For software quality assurance teams: Improve review processes by using subject-typed claim licensing to reduce reviewer bias when assessing evidence for component claims in composite systems.

Authors

Jian Chen, Zixuan Yuan

Abstract

Modern learned systems increasingly combine learned components with search, repair, or external solvers. Benchmarks often measure the resulting end-to-end system, while scientific claims may concern only one component, creating an attribution problem: evidence can fail to support the requested component-level claim while still supporting a positive conclusion about the larger system. Existing evidence-to-claim methods primarily calibrate claim strength. We argue that composite systems require a second dimension: scientific subject. We address this problem with subject-typed claim licensing, which separates weaker conclusions about the requested subject from positive but non-substitutive credit about another subject. We instantiate this idea in SCOPE-Routing for preference-conditioned multigraph routing. Non-authors reproducibly apply the declared semantics; held-out review yields fewer reference-relative upward deviations than unstructured review, while the difference from a strong evidence checklist remains unresolved; and a controlled routing study shows that score-optimal and claim-eligible methods can differ while valid hybrid-system credit is preserved. These results motivate treating claim strength and scientific subject as distinct dimensions of evidence-based evaluation.
View PDFOpen arXiv