Algorithm reduces data traffic for compressed machine learning models
Linear-Time Non-Local Codec Placement for Machine Learning Model Compression
Machine Learning
Summary
Running machine learning models on devices like phones needs smaller data sizes to save space and speed up work. But adding compression parts incorrectly can actually slow things down by creating extra data traffic. The authors created a quick method to place these compression parts in a smart way inside the model’s flow, reducing unnecessary data movement. This helps models run more efficiently on hardware with limited resources.
What this means in practice
- •For mobile app developers: Improve app responsiveness by lowering data traffic during model inference using optimized codec operator placement.
- •For embedded system engineers: Design efficient compressed models by inserting codecs at optimal points to reduce memory and bandwidth costs on hardware.
Authors
Matteo Spallanzani
Abstract
On-device inference imposes constraints on model storage and operand traffic and thus benefits from compressed operand representations, which mandate the use of encoding and decoding operators. However, naïve placement of codec operators can cancel the operand traffic gains sought from compression. In this work, I present an algorithm to place codec operators into a source computational graph to make it compatible with target hardware. The edges of the graph are colored so that each induced subgraph is weakly connected and has a maximal node that both postdominates all subgraph inputs and dominates all subgraph outputs; this unique node - dubbed the $\textit{absolute dominator}$ - is the natural location to observe the traffic flowing through the subgraph and learn parametric codec operators. The algorithm's time complexity is linear in the source graph's size. With respect to naïve local codec placement, the algorithm reduces modeled operand traffic by $24\%$-$44\%$ across four models, including $24\%$ on SigLIP and $34\%$ on SmolLM2.