Replica aware recovery lowers latency in split inference on IoT devices
Replica-Aware Split Inference for Edge IoT Systems
Distributed, Parallel, and Cluster ComputingArtificial IntelligenceSoftware Engineering
Summary
Devices in the Internet of Things are small and can't run big AI models alone, so they split the task across many devices. When one device fails, the process stops working. The paper shows a way to recover by copying the failed device’s model parts onto others, choosing hosts and communication methods to keep the whole process fast. They tested this using a real AI model and found their approach reduces delays better than other common methods.
What this means in practice
- •For iot system engineers: Deploy split AI models that quickly recover from device failures by optimally placing replicas to reduce processing delays.
- •For edge network operators: Improve reliability and speed of machine learning tasks distributed over constrained IoT networks facing device loss or link disruptions.
Authors
Zied Jenhani, Nasrin Razmi, Admela Jukan
Abstract
Split Inference (SI) addresses the hardware resource limitations of Internet of Things (IoT) devices by distributing tiny Machine Learning (TinyML) model segments across multiple IoT devices. However, a device failure can interrupt the inference pipeline. This paper proposes a replica-aware recovery approach that deploys replicas of TinyML model segments originally hosted on failed devices onto surviving devices. We formulate the joint selection of replica hosts and communication protocols as a mixed-integer linear programming (MILP) problem that minimizes post-reconfiguration inference latency, accounting for computation and transmission latencies, under flash memory and runtime memory constraints. Simulations using MobileNetV2 on heterogeneous IoT networks compare the proposed approach with four baselines. We implement the model in a network of 15 IoT devices with a maximum link-rate reduction of 75%, and show that MILP approach achieves an average modeled end-to-end inference latency that is 6.8% lower than that of Computation Greedy and 18.3% lower than that of Random. Index Terms TinyML, split inference, Internet of Things, fault tolerance, replica placement.