Pair counting can exaggerate transformation audit coverage risks
Interpolation Is Not Invariance: Pair Count Is Not Coverage in Transformation Audits
Machine LearningArtificial Intelligence
Summary
When testing how well systems handle changes, counting pairs of examples that look the same is common but can be misleading. The authors explain that many pairs are linked in ways that make the test seem stricter than it really is. They introduce new ways to measure coverage using graph theory and statistics to better understand and limit errors when deploying these systems. Their work helps improve how we check if models truly handle all types of changes without hidden risks.
What this means in practice
- •For machine learning engineers: Identify and better quantify gaps in transformation auditing to reduce deployment risks of models under data shifts.
- •For data quality teams: Evaluate transformation audit coverage more accurately to certify model reliability on diverse data before production.
A theory result. No direct application yet.
Authors
Mohammed Ahnouch, Lotfi Elaachak
Abstract
Counting equivalent pairs is a common way to report transformation-audit coverage, but it can substantially overstate the constraints imposed by an audit: pairs generated from the same semantic object are correlated, and complete orbit graphs contain algebraically redundant edges. We therefore distinguish four complementary quantities---edge count $m$, effective contrast rank $s$, population support rank $r$, and graph spectral gap $η$---and characterize their roles in audit coverage and deployment reliability. Under a rank-$r$ Gaussian contrast model, a population-invariant calibrated reader exists exactly when the anchor has a component in $\ker T$. When an audit has rank $s < r$, its unobserved risk is $R^\star/U$, with $U \sim \operatorname{Beta}((r-s)/2,s/2)$; when $s \ge r$, exact calibrated interpolation is infeasible. The same distinction appears in orbit topology: a spanning tree imposes the same exact-null constraints as a complete graph, while a sharp graph Poincare inequality propagates edge-level drift to an entire orbit at a cost proportional to $1/η$. Cyclic audits can additionally yield zero pair-level leave-one-out error without holding out any semantic object. To address these failures, we derive exact block-Woodbury leave-one-orbit-out updates and introduce a source-disjoint deployment gate over finitely many candidate readers. The gate retains the original reader unless uncertainty bounds certify lower drift within a prescribed clean-utility budget. etc..