Deployed AI system data may mislead evaluation efforts
When Compliance Data Masquerades as Evaluation: Measurement Validity for Deployed AI Systems
Machine Learning
Summary
Sometimes data collected to check if an AI system is working properly gets mistaken for data that compares different AI systems. The authors show that this mistake happens a lot, especially in self-driving cars. They explain why data gathered for safety monitoring can't be used directly to judge how well different AI systems perform compared to each other. They suggest a clear agreement on what the data means before using it to make performance comparisons.
What this means in practice
- •For automated vehicle teams: Design evaluation plans that clearly state assumptions before using operational data as evidence of AI safety performance.
- •For regulatory compliance officers: Avoid drawing unsupported safety claims from compliance data by understanding its limits for comparing deployed AI systems.
A position paper. It proposes an approach and reports no results.
Authors
Hung-Yu Lin, Xingran Huang, Qiming Guo, Jinwen Tang
Abstract
We argue that a recurring failure in the evaluation of deployed AI systems occurs when data collected for operational monitoring or regulatory compliance are interpreted as if they were designed for comparative evaluation. Automated driving provides a concrete example of this problem. U.S. disengagement and crash-reporting regimes produce valuable operational evidence, but differences in reporting scope, exposure, deployment domain, event capture, and comparator construction limit the safety claims that can be supported from these measurements alone. We frame this issue as a measurement-validity problem in AI evaluation rather than as a transportation-specific data limitation. We argue that comparative claims about deployed AI systems require alignment between the intended capability, measured outcome, exposure opportunity, deployment domain, data-generation process, and evaluation comparator. Using automated-driving safety evaluation as a case study, we propose an evaluation contract that makes these assumptions explicit before operational data are interpreted as evidence of comparative performance. The broader implication is that data useful for monitoring deployed AI systems are not automatically valid benchmarks for evaluating them.