LUT-aware transformer speeds up FPGA real-time processing
LEAF: LUT-aware Efficient Attention on FPGAs
Machine Learning
Summary
Making machine learning models called transformers run very fast on special computer chips called FPGAs is hard because the math inside transformers is complicated. The authors created a new method called LEAF that changes parts of the transformer to fit the way FPGAs work best. This method makes the model use fewer hardware resources and run much faster while keeping good accuracy. Their approach is especially useful for tasks that need results in just nanoseconds, like certain physics experiments.
What this means in practice
- •For fpga design teams: Build ultra-low-latency transformer inference engines optimized to minimize lookup table usage for real-time event classification tasks.
- •For real-time data processing engineers: Implement fast transformers that meet sub-microsecond latency requirements for environments like particle physics experiments.
Authors
Arianna Cox, Chang Sun, Lauri Laatu, Benedikt Maier, Aadeesh Sharma, Leo Rozanov, Wayne Luk, Maria Spiropulu, Alexander Tapper, Zhiqiang Que
Abstract
We introduce Lookup-table (LUT)-aware Efficient Attention on FPGAs (LEAF). LEAF is the first nanosecond-scale LUT-aware transformer for FPGAs. Deploying transformers that are large enough to perform well, small enough to fit on an FPGA, and fast enough to meet the sub-microsecond timescales required by real-time systems such as those at CERN remains difficult, since the same attention mechanism that gives these models their high expressivity also makes them costly to implement in hardware. LEAF converts the query, key, and value projections and the final output feed-forward layer into FPGA-native layers with LUT-aware training, creating the first LUT-aware transformer and linformer layers built for ultra-low-latency inference. Attention resists LUT-aware training because it multiplies two activations where the method needs a constant weight to tabulate; LEAF tabulates the attention block's constant matrix-vector multiplications and leaves the rest in arithmetic form. On a standard jet tagging benchmark this improves accuracy by 1.8 percentage points while using 2.9 times fewer LUTs than the previous state-of-the-art sub-microsecond transformer at the same input size. Our most efficient LEAF-based linformer with 64 input particles reaches 80.4% classification accuracy at 28k LUTs and 35ns.