In-sensor vision reduces data and energy use with smart compression
Hardware-Aware Learned Representation Compression for Distributed In-Sensor Vision
Machine LearningArtificial IntelligenceHardware Architecture
Summary
Sending full images from cameras to other devices uses a lot of power and bandwidth. The authors created OASIS, a system that compresses image information inside the camera chip before sending it out, using a tiny neural network. This compression keeps only important details for specific tasks, like recognizing objects, and uses clever coding methods to shrink data even more. Their approach cuts communication needs drastically and lowers energy use, while still keeping accuracy high on tasks like hand and eye tracking.
What this means in practice
- •For embedded systems engineers: Build low-power vision systems by embedding neural compression on sensor chips to reduce data transmission while maintaining task accuracy.
- •For smart camera manufacturers: Incorporate compressed feature extraction in camera hardware to enable efficient real-time analytics without heavy bandwidth or energy costs.$Commercial implications: Allows selling cameras with on-board intelligence that deliver compact, task-focused data streams for real-time applications.
Authors
Chengwei Zhou, Abu Masum, Xuming Chen, Mehran Moghadam, Sreetama Sarkar, Arnab Sanyal, Md Abdullah-Al Kaiser, M. Hassan Najafi, Sercan Aygun, Gourav Datta
Abstract
In-sensor computing reduces the cost of transmitting high-resolution image data by performing early-stage processing near the sensor. However, the logic chip integrated with a CMOS image sensor (CIS) is tightly constrained in compute and memory, limiting conventional deep neural network partitioning. We present OASIS, a distributed in-sensor vision framework that uses a lightweight encoder to generate compact, task-relevant representations before off-chip transmission. The encoder is trained end-to-end using task, entropy, and reconstruction objectives, while the decoder is used only during training. OASIS supports two complementary deployment paths. The first applies 4-bit quantization and Huffman coding while preserving the spatial structure required by classification and dense-prediction tasks. The second uses Sobol-based hyperdimensional computing (HDC) to transform the encoder latent into a fixed-dimensional binary hypervector for associative-memory classification. For the SwinViT-based VWW model, mapping a $3\times3\times8$ latent to a 64-dimensional hypervector provides an additional $1.77\times$ communication reduction with less than one percentage point of accuracy loss relative to the 128-dimensional configuration, yielding an overall $18{,}816\times$ reduction compared with raw 8-bit image transmission. We implement the digital near-sensor pipeline on an AMD Xilinx Zynq UltraScale+ FPGA and characterize it using direct board-level power measurements and Vivado post-implementation analysis, together with circuit-simulated CIS models and a 7-nm ASIC projection. Across visual wake-word classification, hand tracking, and eye tracking, OASIS reduces total system energy by approximately $2\times$-$4.5\times$ while maintaining competitive accuracy, demonstrating a practical hardware-algorithm co-design path for communication-efficient in-sensor vision.