Papers for

big data analytics teams

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

Huffman coding runs faster with less memory in parallel computing

Massively Parallel Algorithms for Huffman Coding

Abstract: Huffman coding is one of the oldest and most fundamental problems in computer science. Given a string of length $\TextLength$ over a general alphabet, the goal is to assign a binary codeword to each character so that no codeword is a prefix of another and the total encoded length of the string is minimized. Huffman coding is widely used in practical compression systems, including file compression. As modern datasets continue to grow, it is natural to study whether a Huffman code can be constructed efficiently in the massively parallel computation (\MPC) model. The celebrated Huffman coding algorithm admits two straightforward \MPC implementations: for any constant $ε\in(0,1)$, one uses $O(\TextLength^ε)$ memory per machine but requires $Θ(\log \TextLength)$ rounds, while the other runs in $O(1)$ rounds but requires $Θ(\sqrt{\TextLength})$ memory per machine. We give the first nontrivial \MPC algorithm for Huffman coding that bypasses both limitations. For every constant $ε>0$, our algorithm uses $O_ε(\log\log \TextLength)$ rounds and $\softO(\TextLength^ε)$ memory per machine, while its total memory and total computation are $\softO(\TextLength)$. This provides a rare example in which an exact solution to a problem whose classical algorithm is sequential in nature can be obtained in a sublogarithmic number of \MPC rounds.

Tue 6 OctData Structures and Algorithms
The gist
Compressing data efficiently is important, and Huffman coding is a classic method to do this. Traditional algorithms to create Huffman codes work step-by-step and can be slow for very large data. The paper shows how to build Huffman codes quickly using many computers working in parallel, while using only a small amount of memory on each machine. This new method speeds up the process by using clever algorithms that run in far fewer steps than before.
Open → 2610.09157v1