Model throughput prediction improves with GGUF metadata across systems
GGUF-Metadata Prediction of Single-Sequence llama.cpp Throughput Across Three Systems
Artificial IntelligenceMachine LearningPerformance
Summary
Predicting how fast AI models run on different computers helps improve their efficiency. This paper shows that using detailed GGUF metadata and tailored prediction methods gives better speed estimates than simpler methods like counting all model parameters. The authors tested their approach across three computer systems and found it works well, though some differences across systems remain. This helps developers understand and optimize model performance more accurately.
What this means in practice
- •For machine learning engineers: Estimate AI model decoding speed accurately on different hardware using GGUF metadata and fitted predictors to optimize runtime configurations.
- •For hardware performance analysts: Compare model throughput predictions across CPUs and GPUs to guide hardware selection and tuning for large language model inference.
Authors
Xinyu Qiu, Chuhong Xu, Bo Su, Ziyao Chen, Ruiyang Xu, Shimeng Dai
Abstract
We predict single-sequence model throughput from GGUF metadata using roofline-shaped predictors with quantization-specific scale factors fitted on reference models. The scored cohort comprises 318 phase-depth measurements from 53 host-file configurations on two Apple M4 Max systems and an NVIDIA RTX 5080. On host-specific held-out sets of four, five, and two configurations, an active-parameter decode model obtains 13.1%, 14.4%, and 36.1% mean absolute percentage error (MAPE), versus 49.4%, 55.3%, and 51.9% when charging total parameters. Leave-one-host-out coefficients fitted on the other two systems yield 11.6%, 16.8%, and 36.0% test MAPE. A low-bit model ladder changes ordering across runtime stacks. The P2 prefill baseline gives 18.7%, 22.2%, and 108.2% test MAPE. GGUF structure helps on all three systems, but fitted efficiencies are not universal.