Papers for

data backup developers

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

Data deduplication achieves strong compression with small edits allowed

Asymptotic Analysis of Data Deduplication Under a Constant Edit Model

Abstract: Data deduplication has become increasingly important in large-scale storage systems, driven by the rapid growth of digital data. By identifying and removing repeats in the data stream, deduplication saves storage space more efficiently than traditional compression methods. In this paper, we study the performance of variable-length deduplication and multi-chunk deduplication algorithms from an information-theoretic perspective, focusing on data streams where repeats are approximate rather than exact. We consider two probabilistic source models in which source strings consist of repeated blocks that are modified by either a constant number of substitution edits or burst-insertion edits per block, the latter capturing the localized edit patterns commonly observed in real-world data. For both models, we show that variable-length deduplication achieves asymptotically arbitrarily large compression ratios, and that multi-chunk deduplication achieves compression within a constant factor of the source entropy under mild conditions.

Fri 6 NovInformation Theory
The gist
Data deduplication is a way to save storage space by detecting and removing repeated parts of information. This study looks at how well deduplication works when parts of the data are similar but not exactly the same because of small changes. The authors find that some deduplication methods can still compress data very efficiently even if there are a few edits in repeated blocks. Their results apply to realistic situations where errors or changes happen in bursts within data.
Open → 2611.09842v1