Study contracts help verify AI research outputs with bound choices
What Output-Only Review Cannot Verify: Study Contracts for Research Agents
Artificial Intelligence
Summary
Checking AI research results just by looking at the output isn't always enough to find errors. The authors show that linking what was promised before the study, what was done, and what can be claimed helps catch defects in AI experiments. They test their approach with known errors and find that verification works better when judges have full information. This points to the need for honest, locked-in contracts and thorough checks to improve trustworthy AI research.
What this means in practice
- •For ai research developers: Use study contracts to formally bind AI experiment steps and detect errors by comparing approved plans with execution evidence, improving trustworthiness in AI studies.
- •For software quality assurance teams: Adopt contract-based verification to systematically check software test outcomes against pre-approved requirements to find subtle faults missed by output-only reviews.
Authors
Eitan Waks, Ben Glocker
Abstract
Some defects in an AI-generated study can be identified from its artifacts; others require knowledge of what was approved before execution. We propose study contracts that bind declared experimental choices, run obligations and claim scope to recorded execution evidence, and distinguish this contract-relative verification from scientific truth. A diagnostic using eight self-authored clean/mutated pairs illustrates the information boundary. A deterministic checker applying a registered, fault-specific rule to approved and executed objects detected all eight registered mutations. Across eighteen recorded judge aliases given individual metadata-filtered packages without pair context or the registry-selected fault label, 104 of 144 mutated evaluation cases received defect flags; the remaining cases comprised 32 abstentions and eight terminal failures, with no explicit clean decisions on mutated cases. Some packages retained approval and execution fields, including digests. The prompt instructed judges to abstain when evidence was insufficient. These results characterize a deliberately information-asymmetric development setting; they do not isolate the effect of authoritative information from differences in task specification and rule selection, and they are not comparative verifier quality or agent reward hacking. We identify full-information comparisons, legitimate-adaptation controls and closed-loop agent evaluations as necessary tests of whether contract checks improve useful compliant completion under optimization.