Spark enhancement reduces memory load during large distributed data aggregation
TreeRedux: Separating Concerns in Spark's Distributed Tree Aggregation
Distributed, Parallel, and Cluster Computing
Summary
Apache Spark is a system that helps process large amounts of data by breaking the work into pieces across many computers. Normally, a single computer called the driver handles the final step of combining results, which can cause it to run out of memory when the processed data is very big. The authors found a way to do some of this final combining work on other computers in the cluster, reducing the memory needed on the driver. This lets Spark handle much larger datasets without crashing from memory problems.
What this means in practice
- •For big data engineers: Handle larger distributed data aggregations in spark clusters without driver memory failures by offloading final aggregation steps.
- •For database administrators: Improve reliability for exact quantile and heavy-hitter queries over massive datasets by reducing coordinator memory pressure in spark jobs.
Authors
David A. G. Harrison, Ivan Cao
Abstract
By default, Apache Spark's tree aggregation primitives place the tree root on the driver, requiring the driver to participate in the same aggregation computation over intermediate aggregation state as executor nodes. For large aggregates, this can expose the single coordinator to substantial computation and memory requirements. Recent Spark versions optionally move the root to an executor, but the completed aggregate must still be returned to and materialized on the driver. We demonstrate this limitation using exact quantile computation and heavy-hitter identification, where the intermediate aggregation state can be substantially larger than the desired final result. We propose TreeRedux, a minimal extension that adds a terminal finalize operation, executed on an executor, that maps the aggregation state U to a compact result V, so that V rather than the potentially large U is materialized on the driver. For exact quantile computation, applying TreeRedux to GK Select removes the driver's epsilon-n memory term, reducing driver memory requirements to the same asymptotic order as Spark's GK Sketch. In our experiments, the default GK Select implementation encountered a driver out-of-memory error at 2.5 billion elements. Spark's executor-side final aggregation option extended this limit to approximately 16-18 billion elements but still required the final aggregation state to be materialized on the driver. Redux Select completed through 28 billion elements without a driver out-of-memory error. TreeRedux allowed Space-Saving sketches with up to 32x the capacity of the largest configuration that materializes a full sketch on the driver.