Clinical coding accuracy improves by modeling coder style variation
Decomposing Error and Style in Automated Clinical Coding
Computation and Language
Summary
Clinical coding automatically assigns diagnostic and procedure codes to patient records, but different medical coders often disagree about which codes to use. The authors found that much of what looks like errors by automated coding systems are actually differences in coding style between teams or sites. By modeling these style differences explicitly, they improved coding accuracy significantly across multiple datasets. So, coding style is not just random noise but a systematic factor that helps explain variation in coding results.
What this means in practice
- •For hospital data teams: Improve automated clinical coding accuracy by incorporating coder style differences tailored to specific sites or teams.
- •For healthcare ai developers: Design clinical coding systems that adjust predictions based on style rubrics to boost performance across varied hospital datasets.
Authors
Han-Chin Shing, Jack Moriarty, Ryan Ware, Afton Marchbanks, Carlyn Canvasser, Stefanie Higgins, Harsh Gupta, Fang Wang, Joseph Paul Cohen
Abstract
In automated clinical coding, where the label space spans tens of thousands of diagnosis and procedure codes, models are currently evaluated against a single gold annotation, treating any deviation as error. But we find when two teams code the same 110 ACI-Bench encounters, they agree on only 73% of codes (Jaccard similarity) for the same note; even after an independent clinical audit removes erroneous codes, agreement rises only to 77%. Is that gap error or something systematic? We model the systematic component as coding style $ψ$, a coder- or site-specific policy over what to code and how much to document, and recast coding as $p(\mathrm{code}\mid\mathrm{note},ψ)$, estimating $ψ$ with a 10-dimension rubric. If style were noise, conditioning on it would do nothing. Instead, across five datasets a model conditioned with a data-matching style raises ICD F1 by up to 26 points and an extreme mismatched one lowers it by up to 21. Four prompt based coding methods spanning 39-49 F1 converge to 52-56 once style is supplied (All p<0.05). Much of what single-gold evaluation charges to model error is recoverable, unmodeled style.