SpBench framework benchmarks multiple data lake engines continuously
SpBench: An Agent-Native Framework for Cross-Engine Lakehouse Benchmarking
DatabasesMachine LearningPerformance
Summary
It's hard and expensive to compare how different data processing systems perform on big data tasks. The authors designed SpBench, a tool that can run standard and custom tests on many big data platforms automatically and all the time. It helps spot performance problems early and guides improvements. SpBench creates fair tests, collects detailed results, and lets smart agents analyze those results to find issues faster.
What this means in practice
- •For data engineering teams: Continuously monitor and compare performance across multiple data processing engines to detect regressions and guide optimizations.
- •For cloud platform operators: Provide standardized benchmarking as a service to customers across different execution environments using an agent-assisted framework.$Commercial implications: Enables selling benchmarking tools that automate cross-engine performance evaluation and regression detection for cloud customers.
Authors
Lyndon Shi, Nic Crouch, Yi Hu, Zihao Ye, Choden Konigsmark, Eric Vandenberg, Dionysios Logothetis, Kuba Niemiec, Krzysztof Gadomski, Yuxiong He
Abstract
SpBench is an agent-native framework for cross-engine lakehouse benchmarking, in production use for benchmarking Snowpark, Snowpark Connect for Apache Spark, and other execution environments across standardized suites -- such as TPCx-BB and TPC-DI -- and custom benchmarks distilled from production workload patterns. It runs continuously across a broad portfolio of workloads, guiding the performance roadmap and detecting regressions for production-scale data engines. Benchmarking has historically been an expensive investment requiring continuous cross-functional collaboration and domain expertise, making it difficult to obtain actionable performance insights. SpBench provides (i) a typed, code-generated benchmark substrate that compiles platform-agnostic benchmark definitions into fair, comparable platform-specific implementations; (ii) a structured-telemetry contract recording every run as queryable fact and dimension data, letting LLM agents query results deterministically; and (iii) an agent-native harness of three composable skills to expedite building benchmarks and analyzing results. This agent-native architecture enabled the rapid development of more than 400 benchmark queries that run on a regular cadence to detect performance regressions; catching these has directly led to improvements in production.