Policy support system improves evidence and audit controls for decisions
Policy-as-Skill: Governed LLM Decision Support with Evidence, Deterministic Control, and Audit
Artificial IntelligenceSymbolic Computation
Summary
Many organizations use large language models (LLMs) to help make decisions about policies, risks, and compliance, but they need ways to check evidence and keep track of versions and reviews. The authors created Policy-as-Skill (PaS), a system that packages policies into executable and versioned modules, helping with evidence validation, review routing, and audit trails. Their system, tested on many tasks, showed better accuracy and audit completeness than some existing methods, and can add rule-based control selectively to improve results further. This approach aims to make decisions supported by LLMs more reliable and transparent.
What this means in practice
- •For compliance teams: Provide automated policy decision support with built-in evidence checks, version tracking, and audit trails for regulatory compliance.
- •For operations teams: Integrate modular policy execution with review routing to streamline operational decision reviews and ensure traceability in workflows.
Authors
Kabeh Mohsenzadegan, Vahid Tavakkoli, Kyandoghere Kyamakya
Abstract
Organizations increasingly use LLMs for policy, compliance, risk, and operational decision support, requiring evidence validation, review routing, version control, and auditability. We introduce Policy-as-Skill (PaS), a modular runtime that packages these functions as executable, versioned policy capabilities. Thirteen methods are evaluated with a fixed Gemma4 backend on 600 development tasks. PaS+Audit achieves 53.8% exact accuracy, macro-F1 0.346, review F1 0.854, citation precision 1.000, policy-reference recall 0.984, and audit completeness 1.000, outperforming LLM+RAG on most governance and review metrics. Deterministic control raises aggregate accuracy to 61.2% but is strongly task dependent, supporting selective rather than universal rule-based intervention.