Benchmarking tool measures model accuracy latency and energy across devices
HoliBench: A Cross-Platform Benchmarking and Deployment Toolkit for Foundation Models in CPS-IoT Applications
Distributed, Parallel, and Cluster ComputingMachine LearningPerformance
Summary
Choosing how to run AI models on small devices like sensors or robots is tricky because speed, power use, and accuracy all matter. Current tools either focus just on accuracy or only work on specific hardware, so people struggle to compare and decide. The authors created HoliBench, a tool that measures all these factors together across many devices, helping users see trade-offs in one place. It makes it easier to pick the best model for limited devices without testing every possible setup.
What this means in practice
- •For embedded system developers: Compare and choose foundation models for devices with limited power and speed by using a single tool that measures accuracy, latency, and energy on many hardware types.
- •For cloud infrastructure engineers: Predict performance and power use of multi-model AI pipelines on heterogeneous servers to optimize deployment without exhaustive testing of all configurations.
Authors
Inesh Chakrabarti, Zejun Xiong, Pragya Sharma, Mani Srivastava
Abstract
Foundation models, including large language models, vision-language models, and time-series foundation models, are increasingly deployed on embedded and edge platforms for CPS and IoT applications, where energy, latency, and memory are as critical as task accuracy. Existing benchmarking tools evaluate model capability in isolation, reporting accuracy assuming sufficient compute, while hardware profiling tools remain platform-specific and mutually incompatible. As a result, users lack a unified workflow for making deployment decisions across heterogeneous devices. We present HoliBench, a modular benchmarking and deployment toolkit that jointly characterizes accuracy, latency, and energy across platforms from single-board computers to GPU servers. Its platform abstraction layer calibrates cross-device measurement, and the toolkit supports multiple model modalities, inference engines, concurrencies, and existing evaluation harnesses. An interactive interface exposes constraint-aware configuration selection over a design space that is profiled once and reused across studies. Using HoliBench, we characterize 20 models across 7 device types, 3 quantization levels, 8 inference backends, and over 30 tasks, surfacing tradeoffs that existing tools miss: quantization reduces latency only on hardware with low-precision support, accuracy gains show diminishing returns relative to energy, and for autoregressive workloads, average inference power is approximately constant across output lengths. We further find that single-model profiles compose under sequential co-resident execution. In a multi-model CPS deployment, standalone profiles predict combined-pipeline latency and power within 1.2% and 2.5%, enabling deployment exploration without exhaustively profiling every pipeline configuration. We release HoliBench as open-source infrastructure for deployment-aware evaluation of foundation models.