Papers for

backup software developers

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

Balanced content-defined chunking improves data deduplication performance

Benchmarking Content-Defined Chunking Algorithms for Data Deduplication

Abstract: Data deduplication relies on content-defined chunking (CDC) algorithms to decide where data is split. We observe that most CDC algorithms are modular: a base chunking algorithm fixes the byte-level cut-point condition, while high-level techniques (size normalization, min/max thresholds, caching) layer on top. Existing benchmarks conflate the two, obscuring which design choices actually drive performance. We isolate and evaluate eight state-of-the-art base chunking algorithms across four realistic workloads and four metrics: throughput, deduplication ratio, mean chunk size, and chunk-size variance. Further, we derive new formulas relating algorithm parameters to expected chunk size for AE, RAM, and MII. In doing so, we uncover a longstanding error in the widely-used AE parameterization reproduced across multiple papers and evaluation tools that produces chunks 13-39% smaller than intended and inflates AE's reported deduplication. Across our evaluation, AE is the most balanced base chunking algorithm across all four metrics, RAM's similarity to AE does not carry over to low-entropy data, and MII and PCI show limited advantages over simpler alternatives in storage-oriented workloads. We release all implementations, datasets, and analysis scripts as open source.

Fri 9 OctDatabasesDistributed, Parallel, and Cluster Computing
The gist
Data deduplication is a way to reduce storage by cutting data into chunks and removing duplicates. The authors studied eight popular methods for deciding where to cut data and tested them on different types of data and performance measures. They found that one method called AE is the most balanced in terms of speed, chunk size, and how well duplicates are found. They also discovered and corrected a common mistake in AE’s settings that led to reporting smaller chunks than expected. The authors share all code and data so others can use and build on their work.
Open → 2610.13704v1