AI summaryⓘ
The authors created PolicyKG, a system that reads policy documents in PDF form and automatically turns natural language rules into machine-readable logic and constraints. It classifies sentences as obligations, permissions, or prohibitions, then converts them into formal logic and SHACL shapes to check rule compliance. Their system uses a flexible vocabulary registry to adapt to different domains by swapping this registry instead of retraining the model. They tested it on institutional policies and a lease contract dataset, showing good accuracy when the vocabulary matched well. PolicyKG also guarantees consistent output and flags when higher-order logic might be needed, though this was rare in their tests.
Institutional policyDeontic logicSHACLLarge Language ModelNatural language processingOntologyCompliance checkingNamed entity classificationCross-domain adaptationFleiss' kappa
Authors
Ponkrit Kaewsawee, Chaklam Silpasuwanchai, Chutiporn Anutariya
Abstract
Institutional policies stay in natural language while the systems that check compliance demand machine-readable constraints. Bridging that gap is still done by hand. PolicyKG closes the loop. It is an LLM pipeline that reads a policy PDF, classifies each sentence as an obligation, permission, or prohibition, lifts the label into first-order deontic logic, and emits SHACL constraints. Four stages run on a LangGraph state machine with per-stage validators. The piece that matters most is the Corpus Adapter: a YAML vocabulary registry that grounds LLM predicates in a target ontology. Retargeting to a new domain means swapping the registry, not retraining a model. On the Asian Institute of Technology Policies and Procedures corpus (1,663 sentences, 443 rules), PolicyKG reaches 86.9% deontic classification accuracy (Cohen's kappa = .709). Three annotators independently re-label a 50-item sample and agree at Fleiss' kappa = .844. SHACL shape correctness on a 69-shape subset is F1 = .866. The FOL path handles 79.2% of rules; the rest go through a direct NL-to-SHACL fallback. We audited every one of the 443 rules for second- or higher-order constructs. An automated regex checklist flagged none, and a first-author pass on the 92 FOL-fallback cases confirmed the same. The exact upper 95% Clopper-Pearson bound on the true HOL rate is 0.67%. This is an audit finding for one corpus, not a proof of FOL sufficiency for institutional policy. Swapping the AIT registry for a GDPR registry raises exact property alignment from 1/15 to 11/15 (Fisher's exact p < .001; Cohen's h = 1.53). On the LexDeMod lease-contract benchmark (N = 200), Macro F1 drops to .370 because lease English uses "shall be entitled" for permission -- exactly the vocabulary mismatch registry swap is meant to fix. Repeated runs produce hash-identical SHACL outputs.