Accelerating Data Preprocessing for Efficient Vision Model Inference on Jetson Edge Device
Performance
Summary
The authors focus on improving how quickly edge devices process images for deep learning tasks by speeding up JPEG decoding, which is usually slow and takes up much time. They use special hardware in NVIDIA Jetson devices called the NVJPEG unit to handle decoding more efficiently, alongside using multiple system components at once (CPU, GPU, DLA). They test various setups on different deep learning models and find that their approach can make preprocessing up to 30% faster for large images. Their work shows how using built-in decoding hardware helps improve overall performance and offers advice on optimizing these workflows.
JPEG decodingedge devicesdeep learning preprocessingNVIDIA JetsonNVJPEG unitmulti-instance designGPUDLAResNet modelsbatch size
Authors
Tian Chen, Nawras Alnaasan, Jinghan Yao, Aamir Shafi, Hari Subramoni, Dhabaleswar K., Panda
Abstract
Data preprocessing is a crucial part of deep learning workflows on edge devices. However, decoding data saved in JPEG format is very compute-intensive and occupies a major portion of the preprocessing pipeline. Therefore, increasing the decoding speed is vital for improving overall throughput, especially for inputs with large image sizes, which are often subject to preprocessing bottlenecks. On the other hand, edge devices are equipped with specialized hardware units to accelerate media processing and image decoding. For instance, the NVIDIA Jetson platform possesses a dedicated NVJPEG unit. These units can be used to enhance the performance of the preprocessing pipeline. This paper introduces the utilization of such specific hardware acceleration units for offloading decoding tasks. By combining this with a multi-instance approach, it allows for the parallelization of all compute resources including CPU, NVJPEG, GPU, and DLA in Jetson devices. In this work, we compare various potential pipeline designs. On ResNet18, ResNet50, and ResNet152, three models with different sizes, we evaluate the impact of batch sizes and image sizes, as well as the characteristics of GPU/DLA inference. Finally, a fine-tuning experiment for multi-instance design has been conducted. The multi-instance design with a specific hardware decoding unit involved offers up to 30.02% speedup for large image sizes, compared with the most optimized design without it. Based on these findings, we demonstrate the benefits of using the NVJPEG unit in deep learning workflows and provide guidelines for tuning and optimizing edge inference workflows.