Hardware software methods speed up video object detection with motion vectors
Analytical and Convolutional Neural Network-Based Motion-Vector Propagation for Efficient Video Object Detection
Computer Vision and Pattern Recognition
Summary
Finding and tracking objects in videos quickly and with low energy use is important, especially on small devices like phones. This paper shows two ways to reuse information about movement between video frames to save time and power. One way uses a simple math formula to guess how objects move, working very fast and using little energy. The other way uses a small neural network to improve accuracy but uses more time and power. The authors also improved how these methods run on hardware to make video analysis faster and more efficient.
What this means in practice
- •For mobile app developers: Improve real-time object detection in video apps on mobile devices by reusing video motion data to reduce processing time and save battery.
- •For surveillance system engineers: Build efficient video analytics that track objects with lower latency and power by integrating hardware and software methods for motion data reuse.
Authors
Ashiyana Abdul Majeed, Mahmoud Meribout, Neethu Joseph
Abstract
Continuous video analytics requires accurate localization at low latency within embedded power budgets. This paper presents a hardware-software design methodology that reuses codec motion vectors (MVs) between detector invocations. Two alternative models support translation and scale changes: analytical motion-vector propagation (Analytical-MV) and learned propagation using a convolutional neural network (CNN) (CNN-MV). The learned model uses convolutional operations and independent object updates suited to parallel execution on an edge graphics processing unit (GPU). Analytical-MV combines a harmonic-mean precision-recall score (F1) of 0.909 with a mean end-to-end latency of 9.03 ms and an energy consumption of 0.177 J per frame, yielding the lowest latency and energy among the evaluated configurations. Relative to detection on every frame, it reduces mean latency by 25.9% and energy per frame by 36.4%. CNN-MV offers a different trade-off: its fastest configuration raises recall from 0.871 for Analytical-MV to 0.890 and lowers mean power from 19.64 to 17.32 W, while achieving a latency of 18.42 ms and an energy consumption of 0.319 J per frame. It is therefore useful when recall or operating power is more important than minimum latency and energy. Execution on a deep learning accelerator (DLA) further reduces time-averaged GPU utilization relative to GPU execution. Host-processing optimization substantially improves both latency and energy, demonstrating the value of jointly designing temporal models and their execution pipelines.