Reffine boosts multi-core data analytics performance across workloads
Unifying In-Memory Data Analytics through Sparse Compilation
Distributed, Parallel, and Cluster Computing
Summary
Handling different types of data analysis tasks efficiently on multi-core computers is very challenging. The authors present Reffine, a tool that unifies how data and operations are represented and optimized, allowing it to speed up many kinds of analyses without needing special tweaks for each type. By turning complex tasks into efficient computer code automatically, Reffine runs much faster than similar existing tools for databases, streaming data, and graph processing.
What this means in practice
- •For database engineers: Create faster analytics engines that run diverse queries efficiently on multi-core hardware without specialized implementations.$Commercial implications: This enables development of high-performance commercial analytical databases that can handle varied workloads with fewer custom optimizations.
- •For big data platform developers: Build streaming and graph analytics systems that achieve significant speed improvements through unified compilation techniques.
Authors
Anand Jayarajan, Gennady Pekhimenko
Abstract
As modern data analytics workloads become increasingly heterogeneous and hardware-intensive, achieving efficient multi-core performance across diverse applications remains an open challenge. We present Reffine, a compiler-based in-memory analytics engine that delivers high performance across a broad range of data analytics workloads. Reffine introduces a novel intermediate representation (IR), grounded in relational algebra and sparse iteration theory, that provides a unified abstraction for data and computation. This representation enables workload-agnostic, end-to-end optimizations such as operator fusion and automatic parallelization across diverse analytics applications. We further develop a sparse compiler backend that translates Reffine IR into hardware-efficient imperative code, achieving high multi-core performance without domain-specific implementations. On the TPC-H benchmark, Reffine outperforms the in-memory analytical database DuckDB by up to $24.9\times$ and the state-of-the-art compilation-based database Umbra by up to $3.2\times$. Reffine also achieves average speedups of $18.3\times$ and $47.9\times$ over Polars and NetworkX on streaming and graph analytics workloads, respectively. Source code: https://github.com/ampersand-projects/reffine