Smart glasses adapt to changing scenes using low bandwidth frequency signals
No Frames, Just Frequencies: Bandwidth-Efficient Deployment-Time Adaptation for Smart Glasses
Computer Vision and Pattern RecognitionMachine Learning
Summary
Smart glasses have to adjust their understanding of what they see when conditions change, but their small size and privacy make this tricky. The authors created a way to send only compact frequency information instead of full images from the glasses to another device. This device then adapts its AI model based on this small data, keeping models accurate with much less data sent. Tests showed it works well with different AI setups and tasks like recognizing people or objects.
What this means in practice
- •For wearable device engineers: Improve smart glasses to adapt AI models in real-time with minimal data sent, preserving privacy and saving bandwidth during changing conditions.
- •For mobile healthcare technology teams: Enable privacy-aware adaptive vision in portable health monitoring devices by communicating compact frequency data instead of raw images for improved model accuracy.
Authors
Francesco Corti, Moritz Becker, Olga Saukh
Abstract
Deep neural networks deployed on smart glasses must adapt to changes in the distribution of incoming data while adhering to strict privacy and device constraints. We present No Frames, Just Frequencies, a bandwidth-efficient source-assisted adaptation framework that characterizes distribution shifts using compact frequency descriptors computed on-device. Rather than transmitting camera frames, the glasses compute and send frequency descriptors to a host device that contains a replica of the model. The host device maps the received frequency descriptor to a distribution shift operator that synthesizes out-of-distribution data and adapts a model replica using a pre-defined adaptation protocol. We evaluate the framework on mixtures of ImageNet-C corruptions using convolution-based (ConvNeXtV2-Femto), self-attention-based (DeiT-Tiny), and hybrid (TinyViT-5M) vision models, on person classification (Visual Wake Words), and on object detection (MS-COCO). Our results reveal that adaptation driven solely by the transmitted descriptors recovers $93-95\%$ of the supervised oracle adaptation gain on all three architectures, $82\%$ on Visual Wake Words and $94$--$98\%$ on MS-COCO. Notably, adaptation quality degrades gracefully as the descriptor resolution decreases from $224 $ to $28$ pixels, consistently across architectures, enabling bandwidth reductions from $213\times$ to $784\times$ across the frequency descriptors.