Multi-path network boosts edge server handling for many IoT devices
MANE: A Multi-Path Adaptive Network for Edge Onloading of Deep Neural Networks
Distributed, Parallel, and Cluster ComputingArtificial Intelligence
Summary
When many smart devices try to use a shared edge server to run parts of their AI models, the server can get overwhelmed and slow down. The authors designed MANE, a system that lets the server switch between different versions of a model to balance accuracy and speed during busy times. MANE also helps decide fairly which devices get full service and which fall back on simpler models so the system keeps working smoothly. Tests showed MANE keeps most devices running on time and more accurate than just letting devices do inference on their own.
What this means in practice
- •For edge computing engineers: Manage shared AI inference resources on edge servers serving many connected devices efficiently with improved latency and accuracy balance.
- •For iot system architects: Build smart indoor environments with many devices supported by an adaptive edge-based AI inference framework that prevents server overload.
Authors
Sokratis Nikolaidis, Stylianos I. Venieris, Leonidas Malachias, Iakovos S. Venieris
Abstract
Split computing constitutes a widely used distributed inference approach, where a lightweight head model is onloaded onto the device and a heavier tail model resides on an edge server, leveraging the growing computational capabilities of modern System-on-Chips while alleviating server load. As intelligent indoor environments such as smart offices grow increasingly populated with diverse IoT devices, a single edge server must simultaneously assist multiple devices, each competing for the same shared inference resources. Without a principled mechanism to manage this shared load, the server is quickly overwhelmed, causing latency SLO violations and rendering server-assisted inference ineffective. In this work, we present MANE, a distributed inference framework that equips the server with a multi-path tail architecture, enabling a dynamic accuracy--throughput trade-off at runtime. By introducing a novel multi-path model architecture, a three-stage training scheme featuring a Joint Head Network Distillation loss and a hysteresis-based scheduler with an equitable device-fallback policy, MANE maintains over 80% SLO satisfaction rate where state-of-the-art onloading methods fail completely, while preserving accuracy 6pp higher than on-device alternatives, across up to 40 concurrent devices.