Hierarchical inference cuts delay and data cost in tabular AI streams
Streaming Hierarchical Inference with Tabular Foundation Models
Machine Learning
Summary
Big AI models that work with tables of data often take too long and use too much communication to handle lots of incoming data quickly. The authors came up with a two-level system called HINT that makes quick guesses locally using smaller data memories and only sends uncertain cases to a more powerful AI in the cloud. This approach balances accuracy and speed by deciding when to keep the work local or ask the cloud for help. Tests show that their method finds good trade-offs between making accurate predictions and reducing data transfer.
Tabular Foundation ModelsIn-context learningHierarchical inferenceEdge computingCloud computingApproximate nearest neighborSliding windowUncertainty estimationData streamingCommunication overhead
Authors
Vitor Crista, Afonso Lourenço, Diogo Martinho, Goreti Marreiros
Abstract
Tabular Foundation Models (TFMs) have recently demonstrated strong predictive performance through in-context learning, but their deployment in high-throughput data streams remains challenging due to communication overhead and latency. We propose \textit{HINT}, a hierarchical inference framework that combines edge-based retrieval with cloud-based TFM inference. A graph-based approximate nearest neighbor memory maintained over a sliding window provides local predictions and uncertainty estimates, allowing confident samples to be processed locally while uncertain instances are selectively offloaded, together with their retrieved context, to a cloud-hosted TFM. The framework exposes an offloading threshold and a neighborhood retrieval policy that can be varied to balance predictive performance and communication cost. Experiments show \textit{HINT} consistently identifies favorable trade-offs.