Tool unifies AI inference costs across Kubernetes and provider bills
Who Pays for the KV Cache? Attributing Shared AI Inference Spend Across Kubernetes and LLM Provider Bills
Distributed, Parallel, and Cluster ComputingPerformance
Summary
Organizations often struggle to track who pays for parts of AI computing because the costs come from different places, like their own servers and external AI services. The authors created a tool called unalloc that brings together all these separate bills into one clear report showing who is responsible for what spending. They tested this tool in various scenarios and found that existing billing often leaves big parts of the cost unassigned or double counted. Their work highlights how complex it is to fairly assign AI costs among teams using shared resources.
What this means in practice
- •For cloud infrastructure teams: Combine Kubernetes and API billing data to produce precise AI inference cost allocation reports across shared deployments.
- •For llm service operators: Measure cost shares for multi-tenant AI workloads by comparing different metering rules on shared GPU resources.
Authors
Timothy Urista
Abstract
Organizations pay for AI through disconnected ledgers: Kubernetes allocations for self-hosted inference, gateway logs, and per-token bills from API providers. We present unalloc, an open-source tool that joins OpenCost, LiteLLM, OpenAI and Anthropic cost data into one exact ledger and reports the share of spend with no owner, and use it to study where attribution breaks at the seams between these systems. Five case studies run inference for real or simulate it: a vLLM-style serving simulator with paged KV memory and prefix caching; a PyTorch transformer serving a multi-tenant trace with a real KV cache; tensor- and pipeline-parallel inference on torch.distributed; the unmodified CLI against mock provider APIs; and four downstream use cases. At the seams, in a constructed multi-pod deployment scenario -- one month of synthetic OpenCost allocations, not observed billing data -- owner labels set only on LeaderWorkerSet leader pods leave 66% of that deployment's GPU bill unowned, and the natural fallback key assigns 61% of it to a Helm chart name while the headline unallocated share falls to 4%; enabling every source double counts all gateway spend; and reading one page of a billing API reports a quarter of spend. Inside a shared inference server the metering rule decides who pays: on an NVIDIA H100 running vLLM, a token meter assigns a retrieval-heavy tenant 12-14 percentage points more of the bill than an equal time-share meter at every load tested, while GPU utilization reads 97-99% across configured loads of 2 to 16 requests per second (3.7 to 26.9 completed requests per second; the configured rate counts session-initial arrivals only) and power draw tracks load. Neither meter is a ground truth; we position these results against recent Shapley-based energy attribution. Code, raw data, captured evidence, figures and the paper regenerate from the repository.