AI pipeline teams safely reuse evidence to speed releases
Reusing Assurance Evidence in AI Pipelines: Validity Conditions, a Fail-Closed Release Gate, and a Replay of 714 Public Data Version Control Pipelines
Computer Vision and Pattern RecognitionSoftware Engineering
Summary
Running every check in an AI system again after any change can be costly, so teams try to reuse previous checks. The authors studied when reusing these checks is safe and found that missing some dependencies causes problems. They tested a special gate that blocks most bad cases and ran it on many public AI pipeline changes to see how often errors happen. By adding extra checks on imported code, reuse becomes much safer while still saving most of the work.
What this means in practice
- •For machine learning engineering teams: Reduce redundant checks by safely reusing prior validation results in AI pipelines using digital signatures and declared dependencies.
- •For data science operations teams: Improve pipeline release decisions by detecting undeclared dependencies that cause unsafe reuse of stage outputs.
Authors
Atanu Mukhopadhyay, Shravya Reddy Nalla
Abstract
Rerunning every governance control after every change to an artificial intelligence (AI) pipeline is expensive, so teams reuse earlier results. Reuse is safe only if each control's declared dependencies cover everything that can change its outcome, and only if the release gate checks that reused evidence describes the candidate being released. This article makes both conditions explicit and measures how often the first fails in practice. It defines when a signed evidence record is valid for a candidate and gives sufficient conditions under which change-local re-execution reaches the same release decision as rerunning every control. A reference gate implementing these checks with Ed25519 digital signatures blocks eight of nine manipulation scenarios, against one for a gate that trusts the planner and three for a gate that only verifies signatures; a dependency the policy never declared passes every gate. A replay of 38,083 commits from 714 public Data Version Control (DVC) pipelines measures that failure mode. Reusing stage results according to declared dependencies avoids 83.4% of stage executions, but in 3.3% of commits it reuses a stage whose imported code or referenced configuration changed, and 31% of pipelines are affected. A manual audit confirms the undeclared dependency in 36 of 40 sampled events. Evaluation stages, whose outputs are the evidence, are the most exposed. Adding each stage's static import closure lowers stale reuse to 0.4% while still avoiding 81.6% of executions; sampled shadow re-execution removes far less, and rerunning everything after unexplained changes avoids only 45.3%.