Extending hardware-friendly multi-threshold activations beyond simple cases
Compiling Multi-Threshold Operators for Quantized Non-Monotonic and Fine-Granular Activations in Custom-Tailored DNN Accelerators
Hardware ArchitectureMachine Learning
Summary
Quantizing neural networks helps run them faster on hardware but makes some calculations, like activation functions, harder to do efficiently. The authors found a way to represent more complex activation functions — including those that don’t just go up steadily and that have detailed per-part settings — using multi-threshold operations. They built tools to automatically convert these functions into this form, making it easier to implement them on specialized hardware like FPGAs. Their work also compares hardware costs and suggests this method could help during training neural networks with quantization in mind.
What this means in practice
- •For hardware accelerator designers: Design customized FPGA accelerators that efficiently implement complex, fine-granular quantized activations beyond monotonic functions.
- •For machine learning engineers: Incorporate multi-threshold representation into quantization-aware training to improve neural network robustness and deployment efficiency.
Authors
Christoph Berganski, Felix Jentzsch, Marco Platzner
Abstract
Aggressive quantization of neural networks can expose activation functions as bottlenecks for efficient hardware acceleration. Implementing quantized activations as multi-threshold operations has been shown to be a promising optimization, allowing for efficient hardware implementation of few-bit activations. However, current implementations are limited to monotonically increasing functions, quantization schemes with one-dimensional (per-channel) granularity and a small set of predefined activation-quantizer combinations. We formalize the multi-threshold representation to extend it to non-monotonic functions and fine-granular parameters, i.e., multi-dimensional thresholds. For automated conversion of quantized function compositions into threshold representation, we propose a systematic approach via algebraic rules integrated into the ONNX-based compiler infrastructure. We experimentally validate our solution on a selection of monotonic and non-monotonic functions by extending the binary search based multi-threshold operator that is integrated into the FINN framework for end-to-end synthesis of custom-tailored FPGA accelerators. The cost of hardware support for non-monotonicity and fine-granularity is characterized in comparison to an alternative approach utilizing existing operators via graph-level decomposition. We compare our method to prior work as well as the approach taken by the hls4ml framework. As an outlook, we demonstrate the prospects of using the multi-threshold representation during the training of a neural network as a form of quantization aware training.