Data streaming adapts live between edge devices and cloud servers
Live Adaptation of Dataflow Stream Processing for Hybrid Edge-Cloud Analytics
Distributed, Parallel, and Cluster Computing
Summary
Processing data either entirely in the cloud or only on edge devices can struggle when networks and resources change. The authors created Sluice, a system that lets data processing tasks move dynamically and smoothly between the cloud and edge devices without stopping work. This helps systems keep up with changing conditions while keeping data accurate and fault-tolerant. They built Sluice on Apache Flink and tested it on real and simulated setups, showing it improves processing speed.
What this means in practice
- •For edge computing teams: Shift data processing tasks live between cloud servers and edge devices to maintain throughput despite fluctuating network and resource conditions.
- •For cloud infrastructure operators: Use adaptive task routing to improve performance and resource utilization in hybrid edge-cloud streaming analytics pipelines.
Authors
Yuanli Wang, Prateek Jain, Lei Huang, Le Xu, Vasiliki Kalavri
Abstract
Dataflow streaming systems, like Apache Flink and Spark Streaming, are integral to modern business analytics, providing real-time insights across industries. However, for emerging hybrid edge-cloud analytics pipelines, neither processing everything in the cloud nor static edge offloading can reliably sustain the target rate under dynamic WAN conditions and limited edge resources. In this paper, we explore how these datacenter-centric platforms can be extended to support such hybrid analytics pipelines. Our goal is to maintain full transparency for user applications, while retaining built-in fault tolerance, out-of-order processing, and existing ecosystem integrations. Achieving this is challenging because existing dataflow engines assume homogeneous resources, rely on static task placement, and do not support adaptation under dynamic network and edge conditions. We introduce Sluice, an adaptive data stream processing system for hybrid clouds, compatible with dataflow streaming engines. Sluice's key contributions are: (i) an automatic query rewriter that decomposes applications into independent, fault-tolerant segments, and (ii) a live routing mechanism that shifts computations between edge and cloud without pausing execution while preserving event-time semantics. We implement Sluice on Apache Flink and evaluate it on a physical testbed with Raspberry Pi devices, as well as on a large simulated testbed of 100 VMs. Across five real-world applications, our results show that Sluice seamlessly shifts computations between edge and cloud, while consistently achieving higher throughput than baseline systems.