A Hardware-oriented Approach for Efficient Bayesian Inference Computation and Deployment
2026-07-20 • Artificial Intelligence
Artificial Intelligence
AI summaryⓘ
The authors focus on making Bayesian inference, a way to handle uncertainty in data, run faster on small, low-power devices using embedded GPUs. They found that most of the delay comes from a type of math operation called tensor contractions. By reorganizing how data is stored and processed, and using smart memory-saving tricks, they created faster versions of three specific algorithms used with Hidden Markov Models. They also made an automatic tool that picks the best method for each case. When tested on an NVIDIA Jetson Orin AGX, their approach sped up calculations by 2 to 5 times without changing the results.
Bayesian inferencevariational message passingtensor contractionHidden Markov Modelsembedded GPUssparse arraysmemory layoutautotuningNVIDIA Jetson Orin AGXPartially Observable Markov Decision Process
Authors
Nikola Pižurica, Matteo Risso, Nikola Milović, Alessio Burrello, Igor Jovančević, Conor Heins, Miguel de Prado
Abstract
Bayesian inference provides a principled foundation for reasoning under uncertainty, but its computational cost hinders deployment on resource-constrained edge devices. In this paper, we present a hardware-oriented methodology for accelerating discrete Bayesian inference on commercial off-the-shelf embedded GPUs. We identify that the latency of a broad class of variational message-passing algorithms is dominated by tensor contractions. Our approach restructures the memory layout of these operations using two complementary merging strategies that produce compact, regularly-shaped primitives better suited for efficient GPU execution. We then introduce optional sparse array representations and a tensor-clustering scheme to reduce the memory footprint. We instantiate the methodology and produce optimized variants of three message-passing algorithms for Hidden Markov Models (HMMs), namely variational filtering, variational message passing, and marginal message passing. Furthermore, we complement this with a machine-learning-based autotuner that automatically selects the best-performing algorithmic variant for a given generative model specification. Benchmarked on an NVIDIA Jetson Orin AGX across 770 randomly sampled realistic Partially Observable Markov Decision Process (POMDP) configurations, our implementations achieve speedups of up to 5x, with typical gains of 2-2.5x, while producing numerically identical outputs to the baseline implementations.