Large language models extract eye test data faster with high accuracy
Can large language models unlock discrete data in ophthalmic diagnostic reports?
Computation and Language
Summary
Ophthalmic diagnostic reports often contain important measurements locked in PDFs that are hard to analyze quickly. The authors tested two ways of using a large language model called GPT-4o to automatically pull out this data into structured formats. Both methods were very accurate and much faster than manual extraction. One method ensured perfectly formatted output, while the other achieved slightly better accuracy on the measurements. These results suggest such models could help speed up handling of eye health records.
What this means in practice
- •For hospital data teams: Automatically extract structured information from ophthalmic PDF reports to speed up electronic health record updating and analysis.
- •For medical software developers: Integrate GPT-4o pipelines into diagnostic software to reduce manual data entry for eye test results with high accuracy and consistent formatting.$Commercial implications: Enables development of clinical software modules that convert scanned eye diagnostic reports into standardized databases for hospitals and clinics.
Tested on one dataset.
Authors
Umair A. Zaidi, An-Lun Wu, Wei-Chun Lin, Thomas S. Hwang, Michelle R. Hribar
Abstract
Objective: To assess the accuracy and efficiency of a large language model (LLM) using two prompt strategies to extract structured data from ophthalmic diagnostic PDF reports. Methods: Twenty deidentified reports across four types (Visual Field, OCT Glaucoma Overview, OCT retinal nerve fiber layer Single Exam, and OCT Thickness Map; n = 5 each) were processed using two GPT-4o-assisted pipelines and compared with a reconciled manual ground truth. Schema-Constrained used Structured Output mode with a predefined JSON Schema; Prompt-Only used a detailed instruction prompt followed by Python conversion to JSON. Outcomes were value accuracy, formatting accuracy, and extraction time. Results: Schema-Constrained value accuracy was 100.00% for Visual Field and RNFL Single Exam, 97.45% for Glaucoma Overview, and 98.00% for Thickness Map; Prompt-Only achieved 100.00% across all four report types. Formatting accuracy was 100.00% for Schema-Constrained across all report types and 100.00% for Prompt-Only except RNFL Single Exam (90.14%). Mean extraction time was 56.51 s per report for manual review versus 5.04 s for Schema-Constrained and 4.70 s for Prompt-Only, an approximately 92% reduction. Conclusions: In this small proof-of-concept dataset, general-purpose LLM-assisted pipelines extracted structured data from ophthalmic diagnostic PDFs with high accuracy and substantially reduced processing time. Prompt-Only achieved the highest value accuracy, while Schema-Constrained produced schema-compliant output with 100% formatting accuracy. These complementary strengths support further evaluation of hybrid, validation-aware workflows for research and clinical data abstraction.