CostApp algorithm precisely allocates expenses in large data pipelines
Product Costing in Data Pipelines
Computational Engineering, Finance, and Science
Summary
Large companies spend a lot of money running systems that process data for their products, but figuring out exactly how much each product's data costs is tricky. This is because data can be used multiple times without being used up, pipelines can have complex loops, and the systems are huge. The authors define this problem clearly and introduce a new method called CostApp that breaks down these loops and simplifies calculations to assign costs accurately. Their method can handle very large data systems quickly, while still giving exact cost distributions.
What this means in practice
- •For data engineering teams: Enable precise allocation of operating costs to specific data products in large-scale pipelines, improving cost transparency and pricing strategies.
- •For cloud infrastructure teams: Support efficient tracking and recovery of data pipeline costs across complex, cyclic dependencies to optimize resource usage and billing.
Authors
Li Peng, Jie You, Feng Shi, Yihang Wu, Jianxin Wang, Weiran Liu, Xinyu Peng, Junhua Wang, Lei Zhang, Lin Qu, Xiaohui Yu, Kui Ren, Jinfei Liu
Abstract
Large data-intensive enterprises spend millions of dollars monthly running data pipelines. Apportioning this cost back to individual products, known as product costing, is a prerequisite for data pricing and cost recovery. However, this process is fundamentally more challenging than physical manufacturing costing due to the unique characteristics of data pipelines, including the non-rivalrous nature of data that complicates the costing formulation, the prevalence of cyclic dependencies that distort the accuracy of results, and the massive scale of pipelines that leads to computational intractability. In this paper, for the first time, we formalize the problem of product costing in data pipelines as a manufacturing cost of data products (MCDP) problem to overcome the non-rivalrous nature of data. To solve MCDP, we propose CostApp, a hybrid algorithm that achieves both exactness and scalability, even for cyclic and massive-scale pipelines. CostApp performs a dimensionality reduction by isolating cyclic dependencies via a feedback arc set (FAS). It aggregates costs onto a compact set of key vertices using a scalable iterative approach, transforming the intractable computational problem into a reduced linear system that can be solved exactly and efficiently by a basic matrix-based method. Experiments on both real-world data and synthetic data show that CostApp scales to process a production pipeline with over 28 million entities within an hour while preserving exactness, supporting pipelines up to 10^2 larger than the direct matrix-based baseline.