LLM-Orchestrated Conformance Checking in Stroke Care Without Computer-Interpretable Guidelines
2026-06-08 • Artificial Intelligence
Artificial Intelligence
AI summaryⓘ
The authors developed a new method that uses large language models (LLMs) to check if patient care follows clinical guidelines by reading regular hospital notes and guideline texts, without needing special computer-readable guideline formats. They tested this method in a stroke care unit, automatically analyzing many patient records against 50 guideline rules. The results showed that most patient care (over 86%) met the recommended guidelines. This study shows that it is possible to use AI language tools to assess healthcare quality directly from usual medical documents.
Conformance CheckingClinical GuidelinesLarge Language ModelsStroke CareComputer-Interpretable GuidelinesPatient Care PathwaysEvent LogNatural Language ProcessingTrace Conformance IndicatorHealthcare Quality
Authors
Giorgio Leonardi, Stefania Montani, Manuel Striani, Alessandro Canessa, Delfina Ferrandi
Abstract
Objective: Conformance checking in healthcare seeks to assess whether patient care pathways adhere to clinical guidelines. However, its practical application often depends on the availability of formal, machine-interpretable representations of guidelines, such as Computer-Interpretable Guidelines (CIGs), which are seldom available in real-world clinical settings. Methods: This work introduces a modular framework based on the orchestration of Large Language Models (LLMs) to support medical conformance checking directly from unstructured clinical and guideline texts, without requiring predefined CIGs. The proposed architecture integrates multiple LLMs and supporting components to extract patient traces from clinical discharge letters, identify normative rules from textual clinical guidelines, translate these rules into executable scripts, and compute a Trace Conformance Indicator to quantify compliance within the event log. Results: The framework was implemented and evaluated in the stroke care domain at the neurological ward of Alessandria Hospital. Hundreds of patient traces were automatically extracted from hospital data and assessed against 50 rules derived from the reference guideline. The analysis showed that more than 86\% of the available traces were conformant. Conclusion: The results demonstrate the feasibility of using orchestrated LLMs for practical healthcare conformance analysis. At the same time, the study provides evidence of a high level of adherence to stroke care guidelines at Alessandria Hospital.