PCFBench: A Diagnostic Benchmark for Product Carbon Footprint Estimation
Artificial Intelligence
Summary
The authors studied how well AI systems calculate the carbon footprint of products step by step, not just in the final result. They created PCFBench, a set of tests that break down the problem into smaller tasks to find exactly where AI struggles. Testing popular AI models showed many make mistakes during the detailed steps, which can hide errors and reduce trust in their results. The authors provide this benchmark to help improve AI accuracy and clarity in measuring product emissions.
carbon footprintAI evaluationproduct carbon footprint (PCF)large language models (LLMs)benchmarkontology matchingnumerical extractionmass conservationworkflow decomposition
Authors
Krishna Rao, Andrew Dumit, Shaena Ulissi, Jacob Feintzeig, P. James Joyce, Daniel Frank, Steven Watson, Jonathan Glidden, Gizem Ilayda Dinc, Travis M. Kwee
Abstract
AI systems are being deployed on high-stakes, domain-specific workflows that demand correctness not just in the final output, but at every intermediate step. One such workflow is estimating a product carbon footprint (PCF), the greenhouse-gas emissions attributable to a physical product. AI agents are increasingly being used to generate PCFs, but existing evaluations score either total emissions (hiding error sources and cancelling mistakes) or sub-tasks in isolation (missing compositional interactions). We introduce PCFBench, the first benchmark to carve PCF modeling into independently-evaluable tasks that require decomposition, retrieval, ontology matching, and numerical extraction. It comprises 614 expert-labelled items across six tasks. Together they probe reasoning under under-specification, conflicting context, and numerical constraints. Across eight frontier LLMs from four providers, no single model dominates. Although the strongest models estimate total product emissions within 2 times of declared totals on 77% of products, this rate drops to 37-58% when the PCF is generated step by step, with only 45-75% obeying mass conservation. These failures undermine the transparency practitioners need to compare products and drive decarbonization. We release the dataset and evaluation harness to support targeted progress.