Hierarchical system speeds up predictions on streaming data with edge and cloud
Streaming Hierarchical Inference with Tabular Foundation Models
Machine Learning
Summary
Fast and accurate predictions on data streams are hard when sending everything to the cloud causes delays and extra communication. The authors propose a method called HINT that first tries to make predictions on a local device using a recent memory of past data. When the local system is unsure, it sends just those uncertain cases along with related information to a powerful cloud model for better analysis. Their approach lets users choose how much to rely on the local or cloud parts, balancing speed and accuracy. Experiments show this method finds a good middle ground between quick responses and prediction quality.
What this means in practice
- •For data engineers: Improve real-time data processing systems by reducing cloud calls and speeding predictions through local uncertainty-based filtering.
- •For iot platform developers: Build IoT solutions that balance latency and accuracy by using edge devices to pre-filter data before cloud-intensive analysis.
Authors
Vitor Crista, Afonso Lourenço, Diogo Martinho, Goreti Marreiros
Abstract
Tabular Foundation Models (TFMs) have recently demonstrated strong predictive performance through in-context learning, but their deployment in high-throughput data streams remains challenging due to communication overhead and latency. We propose \textit{HINT}, a hierarchical inference framework that combines edge-based retrieval with cloud-based TFM inference. A graph-based approximate nearest neighbor memory maintained over a sliding window provides local predictions and uncertainty estimates, allowing confident samples to be processed locally while uncertain instances are selectively offloaded, together with their retrieved context, to a cloud-hosted TFM. The framework exposes an offloading threshold and a neighborhood retrieval policy that can be varied to balance predictive performance and communication cost. Experiments show \textit{HINT} consistently identifies favorable trade-offs.