Papers for

big data platform developers

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

Reffine boosts multi-core data analytics performance across workloads

Unifying In-Memory Data Analytics through Sparse Compilation

Abstract: As modern data analytics workloads become increasingly heterogeneous and hardware-intensive, achieving efficient multi-core performance across diverse applications remains an open challenge. We present Reffine, a compiler-based in-memory analytics engine that delivers high performance across a broad range of data analytics workloads. Reffine introduces a novel intermediate representation (IR), grounded in relational algebra and sparse iteration theory, that provides a unified abstraction for data and computation. This representation enables workload-agnostic, end-to-end optimizations such as operator fusion and automatic parallelization across diverse analytics applications. We further develop a sparse compiler backend that translates Reffine IR into hardware-efficient imperative code, achieving high multi-core performance without domain-specific implementations. On the TPC-H benchmark, Reffine outperforms the in-memory analytical database DuckDB by up to $24.9\times$ and the state-of-the-art compilation-based database Umbra by up to $3.2\times$. Reffine also achieves average speedups of $18.3\times$ and $47.9\times$ over Polars and NetworkX on streaming and graph analytics workloads, respectively. Source code: https://github.com/ampersand-projects/reffine

Thu 24 SeptDistributed, Parallel, and Cluster Computing
The gist
Handling different types of data analysis tasks efficiently on multi-core computers is very challenging. The authors present Reffine, a tool that unifies how data and operations are represented and optimized, allowing it to speed up many kinds of analyses without needing special tweaks for each type. By turning complex tasks into efficient computer code automatically, Reffine runs much faster than similar existing tools for databases, streaming data, and graph processing.
Open → 2609.30497v1