Profiling-Driven Adaptive Distributed Transformer Inference on Embedded Edge Deployment
2026-05-25 • Distributed, Parallel, and Cluster Computing
Distributed, Parallel, and Cluster ComputingArtificial Intelligence
AI summaryⓘ
The authors studied how to split Transformer model tasks across small edge devices to save memory and computing power, using real NVIDIA Jetson hardware instead of just simulations. They found that communication between devices is slowed down mainly because data must pass through the CPU before reaching the GPU, which is slower than expected. To fix this, they created a method called Prism that compresses data and decides when to run tasks locally or across devices, based on offline profiling. Their tests showed Prism makes processing faster and uses less energy compared to simply sharing full data between devices all the time.
TransformerEdge devicesNVIDIA Jetson Orin NanoCPU-GPU communicationWiFi networkingNCCLGLOOSegment Means compressionProfilingDistributed inference
Authors
Muhammad Azlan Qazi, Alexandros Iosifidis, Qi Zhang
Abstract
Distributing Transformer inference across embedded edge devices can alleviate individual memory and compute constraints, yet practical benefits on real hardware remain unclear: prior work relies largely on simulations that overlook hardware-specific communication overheads. We present a hardware prototype study on NVIDIA Jetson Orin Nano devices connected over WiFi. Our key finding is that the dominant bottleneck is not just network bandwidth but also the CPU-GPU staging during communication. Because Jetson's integrated GPU architecture lacks the PCIe/NVLink pathway that NCCL requires, all inter-device data communication should be routed through GLOO and staged in CPU memory; an overhead that scales with communication data volume and makes full-tensor exchange slower than single-device inference across the batch sizes for medium sized models such as ViT. We therefore evaluate Prism by combining Segment Means compression with lightweight offline profiling to adaptively select between local and distributed execution at runtime. Experiments show that this strategy reduces latency by 65%-77% and energy consumption by 34%-52% relative to full-tensor exchange in static distributed execution setup, demonstrating that profiling-driven adaptation is essential for practical distributed Transformer inference on embedded hardware.