Empowering On-Device Model Adaptation with an Edge AI Inference Accelerator
2026-07-20 • Machine Learning
Machine LearningHardware ArchitectureComputer Vision and Pattern Recognition
AI summaryⓘ
The authors developed a way to update parts of a neural network directly on small devices without needing heavy computing power. They use a special chip, the Hailo-8L, to run the main part of the network with simpler calculations, while only a small part is adjusted on the device’s CPU. This method speeds up training and uses less energy compared to just a Raspberry Pi CPU. They also found a way to keep accuracy high by fixing some issues caused by simplifying calculations. Their work shows how to efficiently personalize AI models on limited hardware.
on-device trainingneural network adaptationedge AI acceleratorINT8 quantizationfeature extractionfrozen backbonefine-tuningpost-training quantizationenergy-efficient AIHailo-8L
Authors
Mateusz Piechocki, Alessandro Capotondi, Marek Kraft
Abstract
On-device model adaptation is essential to enable lifelong personalization on resource-constrained hardware, but compute, power, and memory limitations of such devices make end-to-end backpropagation impractical for modern deep neural networks. This work proposes a heterogeneous adaptation pipeline that repurposes a commercial edge AI inference accelerator, Hailo-8L, for frozen-backbone feature extraction during on-device training. The computational graph is partitioned so that the pre-trained backbone is quantized to INT8 and run on the accelerator, while only a lightweight FP32 classification head is fine-tuned on the host CPU, enabling frequent, energy-efficient in-field updates with most weights remaining fixed. Across multiple architectures and datasets, this pipeline achieves up to 15.4x faster wall-clock training time compared to a Raspberry Pi 5 CPU baseline, offers competitive throughput in favorable settings, and consistently reduces energy per sample. Post-training quantization restoration is shown to be crucial for preserving the quality of accelerator-generated features and mitigating accuracy loss in quantization-sensitive architectures. Overall, the results demonstrate a practical approach to efficient on-device adaptation using inference-oriented edge accelerators. The implementation is available at https://github.com/MatPiech/accelerator-training.