Closed-Loop Validation-Repair for Healthcare Interoperability: A Multi-Model Study of Schema Compliance in Clinical LLMs
2026-07-27 • Computation and Language
Computation and LanguageArtificial Intelligence
AI summaryⓘ
The authors studied how well three different AI language models follow healthcare data rules needed for things like diagnostic coding and billing. They found that all models made similar types of mistakes, mainly mixing up formats instead of not understanding medical content. By using a system that checks outputs and fixes errors in a loop, the authors improved the models' accuracy to about 99% compliance with healthcare data standards. This method helps AI systems work better with electronic health records by making sure data fits required rules.
healthcare interoperabilityICD-10CPT codesHL7 FHIRlarge language modelsschema complianceclinical reasoningvalidation-repair frameworkelectronic health recordsdata standards
Authors
Jianru Shen
Abstract
Healthcare interoperability requires AI systems to produce structured outputs conforming to standardized schemas including ICD-10 for diagnostic coding, CPT for procedure billing, and HL7 FHIR for data exchange. While large language models demonstrate clinical reasoning capabilities, their integration into electronic health record systems faces a critical barrier: schema noncompliance. We evaluate three open-source models, Qwen2.5 7B, Llama 3.1 8B, and Gemma2 9B, via local deployment across 320 clinical scenarios spanning ten medical specialties, yielding 960 model-scenario pairs assessed under paired baseline and validation-repair conditions. First, schema noncompliance is consistent across the three model families, with baseline compliance rates ranging from 85.9 to 91.6 percent despite varying architectures and training data, suggesting shared gaps in medical training corpora rather than model-specific limitations. Second, 96 percent of validator-detected failures are representation-level format violations such as alternative medical abbreviations and code prefixes, indicating models follow clinical writing conventions but lack awareness of healthcare IT standards. Third, the validation-repair framework achieves 99.0 percent overall compliance, ranging from 98.4 to 99.4 percent across models, with most errors resolving within one or two iterations. Exact McNemar p-values below 0.001 and absolute improvements of 7.8 to 12.5 percentage points across model sizes confirm statistical significance. These results support closed-loop validation-repair as an effective system-level safeguard for healthcare interoperability, improving schema-level readiness for downstream clinical system integration.