To Redact, or not to Redact? A Local LLM Approach to Deliberative Process Privilege Classification
2026-05-11 • Computation and Language
Computation and LanguageArtificial IntelligenceInformation Retrieval
AI summaryⓘ
The authors studied how to automatically identify sensitive parts of government documents that can be withheld under FOIA laws, focusing on the deliberative process exemption. They used small language models that can run locally, avoiding legal issues with sending documents to cloud services. By combining special prompting methods, their approach improved detection of sensitive sentences, especially those showing opinions and first-person language. Their model's accuracy nearly matched a popular commercial system. This work helps making sensitive government document review faster and more private.
FOIAdeliberative process privilegesensitivity classificationLarge Language ModelsChain-of-Thought promptingfew-shot learninglocal model deploymentrecallF2 scoreopinion expression
Authors
Maik Larooij, David Graus
Abstract
Government transparency laws, like the Freedom of Information (FOIA) acts in the United States and United Kingdom, and the Woo (Open Government Act) in the Netherlands, grant citizens the right to directly request documents from the government. As these documents might contain sensitive information, such as personal information or threats to national security, the laws allow governments to redact sensitive parts of the documents prior to release. We build on prior research to perform automatic sensitivity classification for the FOIA Exemption 5 deliberative process privilege using Large Language Models (LLMs). However, processing documents not yet cleared for review via third-party cloud APIs is often legally or politically untenable. Therefore, in this work, we perform sensitivity classification with a small, local model, deployable on consumer-grade hardware (Qwen3.5 9B). We compare eight variants of applying LLMs for sentence classification, using well-known prompting techniques, and find that a combination of Chain-of-Thought prompting and few-shot prompting with error-based examples outperforms classification models of earlier work in terms of recall and F2 score. This method also closely approaches the performance of a widely-used, cost-efficient commercial model (Gemini 2.5 Flash). In an additional analysis, we find that sentences that are predicted as deliberative contain more verbs that indicate the expression of opinions, and are more often phrased in in first-person. Above all, deliberativeness seems characterized by the presence of a combination of multiple indicators, in particular the combination of first-person words with a verb for expressing opinion.