Ai auto-research aims to build trustworthy data systems with verifiable correctness
Toward Verifiable Auto-Research for Data Systems
Databases
Summary
Building custom data systems using AI can be tricky because the AI might optimize the wrong things, like speeding up benchmarks but causing errors or losing data. The authors discuss how vague user instructions can lead to systems that don’t work as expected. They propose a method to make the research process more reliable by using clear, precise requirements and checking correctness as part of building the system rather than after. This approach also aims to reduce the time and cost needed for research while ensuring the systems are safe to use in real life.
What this means in practice
- •For data infrastructure teams: Create tailored data systems for specific workloads that are provably correct and efficient through cost-effective automated research.
- •For cloud platform engineers: Integrate verifiable AI-generated systems to improve reliability and auditability under realistic operational conditions like concurrency and failures.
A position paper. It proposes an approach and reports no results.
Authors
Kaiwen Zheng, Srijan Srivastava, Zheng Yan, Xinpeng Zhao, Shaohua Li, Chieh-Jan Mike Liang, Tianyu Chen, Haoxian Chen, Eric Lo
Abstract
AI auto-research is poised to forge bespoke data systems, from key-value stores and indexes to query engines, each engineered for a specific workload and hardware platform. Yet this promise rests on fragile ground. An agent optimizes what it can measure, not necessarily what users need. Natural-language requirements leave ample room for specification gaming, producing systems that blaze through benchmarks while silently losing updates or violating consistency. Worse, each research iteration consumes tokens, while validating a candidate under real concurrency, memory pressure, and failure conditions can stretch the entire research process into weeks. In this paper, we envision a more cost-efficient and verifiable approach to auto-research. Precise and composable specifications anchor the search, lightweight performance critics prune costly dead ends, and falsification and deductive verification make correctness an integral part of the process rather than an afterthought. By treating correctness, performance, research time, and token cost as equally important objectives, we aim to transform AI-generated data systems from plausible artifacts into trustworthy, production-grade systems whose behavior can be understood, audited, and ultimately relied upon.