Papers for

hardware performance analysts

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

Model throughput prediction improves with GGUF metadata across systems

GGUF-Metadata Prediction of Single-Sequence llama.cpp Throughput Across Three Systems

Abstract: We predict single-sequence model throughput from GGUF metadata using roofline-shaped predictors with quantization-specific scale factors fitted on reference models. The scored cohort comprises 318 phase-depth measurements from 53 host-file configurations on two Apple M4 Max systems and an NVIDIA RTX 5080. On host-specific held-out sets of four, five, and two configurations, an active-parameter decode model obtains 13.1%, 14.4%, and 36.1% mean absolute percentage error (MAPE), versus 49.4%, 55.3%, and 51.9% when charging total parameters. Leave-one-host-out coefficients fitted on the other two systems yield 11.6%, 16.8%, and 36.0% test MAPE. A low-bit model ladder changes ordering across runtime stacks. The P2 prefill baseline gives 18.7%, 22.2%, and 108.2% test MAPE. GGUF structure helps on all three systems, but fitted efficiencies are not universal.

Mon 14 SeptArtificial IntelligenceMachine LearningPerformance

The gist

Predicting how fast AI models run on different computers helps improve their efficiency. This paper shows that using detailed GGUF metadata and tailored prediction methods gives better speed estimates than simpler methods like counting all model parameters. The authors tested their approach across three computer systems and found it works well, though some differences across systems remain. This helps developers understand and optimize model performance more accurately.

Open → 2609.14864v1