Hardwired sensor units make fast neural decisions without a host
Hardwired Distributed Processing Units for Inference: Neural Decisions at the Sensor that Free the Host
Hardware ArchitectureDistributed, Parallel, and Cluster ComputingMachine Learning
Summary
Processing complex decisions from images usually requires sending data to a main computer, which can be slow and costly. The authors present a small, specialized circuit built into a camera sensor that can quickly decide if a zebra crossing is present, without needing a full computer to think about it. This unit is very efficient, works in real-time, and sends only small amounts of data to the host device. Their work shows that smart decisions can be made right where images are captured, reducing workload on bigger systems.
What this means in practice
- •For automotive system integrators: Implement fast zebra crossing detection directly on vehicle cameras to reduce processing load on central computers and improve real-time response.
- •For embedded device manufacturers: Design low-cost FPGA-based smart cameras that make quick object detection decisions without hosting an operating system or processor.
Authors
Raj Mali
Abstract
We show that a hardwired distributed processing unit (HDPU) can take a neural decision at the sensor and so remove the inference from the host. Built for zebra-crossing detection on a low-cost Artix-7 FPGA, the unit decides on every camera frame in a fixed 2,269,477 clock cycles (34.04 ms depends upon Fpga clk ), inside the 37.49-ms frame period, and on the board it returns the same scores and decisions as its bit-exact software model on all 639 test images. The whole detection path, from camera pixels to decision, takes 1,814 LUTs, with no processor, no operating system and no full-resolution frame store. When the unit answers instead of reporting "not sure", 95.7% of its answers on street photographs are correct, and a hardware contrast stage lifts accuracy on camera-like dashcam frames from 47.7% to 78.2%. The host is spared all of the inference: 8.13 ms of a core in every frame period, where a full-precision ResNet-18 would need more than a frame period even on 16 threads. It also receives 16 bytes per frame instead of 614,400, 38,400 times less. The unit is an 80x60-pixel pipeline feeding a four-layer binarized CNN with 93,408 one-bit weights, verified bit-exactly from a Python model through RTL simulation to the board. Earlier HDPUs did pre-processing (edges, corners, steering); this one shows that the decision itself can move to the sensor.