Papers for

data quality teams

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

Pair counting can exaggerate transformation audit coverage risks

Interpolation Is Not Invariance: Pair Count Is Not Coverage in Transformation Audits

Abstract: Counting equivalent pairs is a common way to report transformation-audit coverage, but it can substantially overstate the constraints imposed by an audit: pairs generated from the same semantic object are correlated, and complete orbit graphs contain algebraically redundant edges. We therefore distinguish four complementary quantities---edge count $m$, effective contrast rank $s$, population support rank $r$, and graph spectral gap $η$---and characterize their roles in audit coverage and deployment reliability. Under a rank-$r$ Gaussian contrast model, a population-invariant calibrated reader exists exactly when the anchor has a component in $\ker T$. When an audit has rank $s < r$, its unobserved risk is $R^\star/U$, with $U \sim \operatorname{Beta}((r-s)/2,s/2)$; when $s \ge r$, exact calibrated interpolation is infeasible. The same distinction appears in orbit topology: a spanning tree imposes the same exact-null constraints as a complete graph, while a sharp graph Poincare inequality propagates edge-level drift to an entire orbit at a cost proportional to $1/η$. Cyclic audits can additionally yield zero pair-level leave-one-out error without holding out any semantic object. To address these failures, we derive exact block-Woodbury leave-one-orbit-out updates and introduce a source-disjoint deployment gate over finitely many candidate readers. The gate retains the original reader unless uncertainty bounds certify lower drift within a prescribed clean-utility budget. etc..

Mon 14 SeptMachine LearningArtificial Intelligence
The gist
When testing how well systems handle changes, counting pairs of examples that look the same is common but can be misleading. The authors explain that many pairs are linked in ways that make the test seem stricter than it really is. They introduce new ways to measure coverage using graph theory and statistics to better understand and limit errors when deploying these systems. Their work helps improve how we check if models truly handle all types of changes without hidden risks.
Open 2609.14870v1