ConvLUT enables ultra-fast convolutional neural networks on FPGAs

ConvLUT: Rethinking Convolutional Architectures for Ultra-Low-Latency LUT-Based Inference

Machine Learning

Summary

Faster computer programs can recognize images and sounds by using special tables called lookup tables, but so far these have mostly worked with simple network types. The authors designed a new way to build more complex networks, like those that analyze pictures and sounds, using these lookup tables. Their new method, ConvLUT, runs extremely fast on hardware called FPGAs and still gives accurate results. This breakthrough shows it’s possible to make super quick and accurate neural networks for real-world tasks like detecting objects or recognizing keywords.

What this means in practice

Authors

Marta Andronic, Olivia Weng, Danial Zuberi, Ryan Kastner, George A. Constantinides

Abstract

Lookup-table (LUT)-based neural networks (NNs) offer a compelling path to ultra-low-latency inference on FPGAs. However, existing LUT-based neural networks have largely been restricted to MLP-style topologies, leaving structured vision and audio tasks underexplored. We present ConvLUT, a family of fully unrolled LUT-mappable convolutional architectures targeting the MLPerf Tiny benchmarks. It is comprised of a standard convolutional model built around a hierarchical tree of LUTs and a depthwise separable model following the MobileNet design philosophy. These architectures are enabled by four key contributions: a unified LUT-based convolutional framework supporting standard, depthwise, and pointwise convolutions under explicit fan-in constraints; a LUT-tree that addresses the $k^2c$ receptive field dimensionality bottleneck; a channel-partitioned LUT pooling layer that replaces conventional spatial pooling; and a hardware-aware iterative pruning strategy for stable convergence at scale. Evaluated on CIFAR-10, Visual Wake Words, and Keyword Spotting, ConvLUT achieves competitive accuracy with fully unrolled inference latencies below $\sim 50$ ns. On CIFAR-10 it reaches at least a $8844\times$ latency reduction over non-logic-based prior implementations, and outperforms existing MLP-style LUT-based models on the same task by at least $20$ pp in test accuracy.