Custom data compression algorithms improve speed and size by two times
The Case for Synthesized Data Compression
Databases
Summary
Compressing data helps save space but common methods are made to work well on all kinds of data, not perfect for any one type. The authors suggest creating special compression programs designed just for each specific dataset. These custom programs can be fully checked for accuracy before use. They also showed their new approach can store data in about half the space or decompress it twice as fast as general methods.
What this means in practice
- •For storage system engineers: Create dataset-specific compression algorithms verified for correctness to improve storage efficiency and retrieval speed beyond standard compression tools.
- •For mobile app developers: Develop apps that store user data using custom compression tailored to app-specific data types for faster access and reduced storage use.
Authors
Ivan Iachnyk, Mihail Stoian, Luisa Neubauer, Ryan Marcus, Andreas Kipf
Abstract
Compression is a fundamental tool in data management. Today's compression schemes are generally designed by hand to apply to many different types of data (e.g., run length encoding or arithmetic encoding), leading to a zoo of general-purpose algorithms which work well for many datasets, but may not be optimal for any specific dataset. Inspired by recent advancements in agentic coding, we propose synthesizing custom-tailored data compression algorithms on a per-dataset basis. Unlike prior work using agents to synthesize database components, a data-specific compression algorithm can be fully verified at construction time, alleviating most correctness concerns. Modern file formats, like AnyBlox and F3, allow including the synthesized code alongside the data itself. Experimentally, we show that our synthesis agent can expand the compression-factor/decompression-speed Pareto front, outperforming general-purpose methods by nearly $2\times$ in compression factor.