Pfas regulatory information extracted accurately with large language models
RegExtract: A Benchmark for PFAS Regulatory Information Extraction with LLMs
Computation and Language
Summary
Legal documents about PFAS chemicals are long and complicated, making it hard to find important rules and facts. The authors created RegExtract, a collection of annotated government documents from 20 US states with key information and sources marked. They tested different large language models to pull out this structured data and found it can be challenging to identify some fields, but providing specific examples helps. This benchmark and evaluation approach aim to improve tools that help companies and regulators understand and follow PFAS rules.
What this means in practice
- •For compliance teams: Extract detailed PFAS regulatory requirements and trace sources within documents to improve monitoring and reporting accuracy.
- •For environmental software developers: Integrate RegExtract data and methods to build AI tools that automate PFAS risk identification and regulatory question answering.$Commercial implications: Enables commercial software with automated PFAS compliance features for manufacturers and environmental consultancies.
Authors
Defne Circi, Krystelle Lionti, Sarathkrishna Swaminathan, Daniel P. Sanders, L. Catherine Brinson, Bhuwan Dhingra, Basel Shbita, Mohab Elkaref, Sara Capponi, Tim Erdmann, Brandi Ransom
Abstract
Extracting structured information from regulatory documents is a critical but challenging task. Relevant facts are scattered across long, legally dense texts, and traceability of extracted facts to their source is essential for verification. With increasing awareness of the environmental and health impacts of per- and polyfluoroalkyl substances (PFAS), regulatory documents have expanded significantly, varying in scope and product relevance, making interpretation challenging for manufacturers and policymakers. We introduce RegExtract, a benchmark for structured information extraction from PFAS regulatory documents, comprising 488 human-annotated field instances across 20 US state documents and 46 regulations. Each field instance includes a value and a provenance entry, yielding 976 entries in total. We define a typed, multi-field schema targeting key elements such as material definitions, permitted uses, and product exemptions, and propose provenance-aware evaluation metrics tailored to regulatory text. Through experiments with open- and closed-source LLMs across prompting strategies, document condensing methods, and schema partitioning approaches, we find that document identification fields remain challenging, and that field-specific few-shot examples improve extraction. This work lays the groundwork for AI tools for hazardous substance identification, question answering, and compliance monitoring. The code and data are available at https://github.com/IBM/PFAS-regulations.