Law of Large Numbers: Accuracy as Statistical Measure for AI Compliance and Competition
2026-08-31 • Computers and Society
Computers and Society
AI summaryⓘ
The authors explore how the word "accuracy" means different things in machine learning and legal contexts, especially under the EU AI Act. While machine learning researchers use accuracy mainly to measure system performance, they also know it has limits for real-world usefulness. Meanwhile, legal frameworks embrace ambiguity in accuracy to stay flexible but still rely on it for compliance. The authors identify five key differences in how accuracy is understood, showing that neither side fully grasps it outside their own context. They suggest research and tools to help clarify and improve how accuracy is measured and applied.
accuracymachine learningEU AI Actperformance metricslegal compliancestandardizationmeasurement validitystatistical evaluationnormative vs empiricalAI regulation
Authors
Rabanus Derr, Alina Wernick, Robert C. Williamson
Abstract
The machine learning community progresses (in part) by improving the "accuracy" of its systems. The EU AI Act explicitly refers to "accuracy" as part of its compliance measures for high-risk AI systems. Are we talking about the same thing? This work presents "accuracy" as a case-study for differing requirements of social worlds, the technological machine learning community and the legal community. While competition on accuracy contributes to technological development, machine learning scholars simultaneously recognize accuracy's shortcomings regarding the usefulness and effectiveness of machine learning systems. The legal counterpart embraces the vagueness of "accuracy," leaving interpretative flexibility for technological and societal changes. At the same time, accuracy is a core element of compliance within the EU AI Act. We elaborate on five main tensions, (a) nature of accuracy, (b) notion of performance, (c) scope of validity, (d) ends, and (e) statisticalness, to show that the two communities project disparate, and sometimes contradictory, expectations on accuracy. Both legal and technical communities lack precise understanding of "accuracy" beyond the contextual boundaries of their community. The resulting frictions, \eg, based on the empirical or normative understanding of accuracy, are symptoms of an unresolved (and unresolvable) debate on what accuracy is. We constructively use the frictions to recommend baselines and interventional studies in standardization, and demand for tools to extend the validity of accuracy measurements.