LLM framework extracts lung cancer tumor information clearly and accurately
SIFTING: A Novel LLM-Based Framework for Structured and Transparent Information Extraction from Clinical Free-Text Reports, with Application to Tumor Staging in Lung Cancer
Computation and Language
Summary
Extracting important medical details from doctors’ notes is tricky because the text is unstructured and hard to verify. The authors created SIFTING, a tool that uses a large language model to pull out tumor stage information from lung cancer reports in a clear and traceable way. Their method links each finding back to the exact part of the text it came from, making it easier to check and trust. Tests showed SIFTING was about as accurate as experts and big patient data models, while giving full control over the data and outputs.
large language modelclinical free-textinformation extractiontumor staginglung cancerradiology reportsstructured datatraceabilityLlamabootstrap resampling
Authors
Mirco Hess, Gerben van Veenendaal, Joris Wakkie, Yiwen Soo, Malcolm H. Lawson, John D. Maclay, Arjun Nair, Neal Navani
Abstract
Background: Large language models (LLMs) show promise for extracting information from clinical free-text documents, but their outputs are often unstructured and lack traceability, complicating validation and adoption in clinical workflows. In this work we introduce SIFTING, an LLM-based framework designed to address these shortcomings. Methods: SIFTING combines the language comprehension capabilities of LLMs with segment-level processing and structured prompts with strict output control, linking findings to the source text to enable both accurate and transparent information extraction. To demonstrate its capabilities, we applied the framework to the task of extracting tumor T-stage information from 130 lung cancer radiology reports (SIFTING-T-stage). A compact 4-bit quantized version of the open-source LLM Llama-3.3-70B (35 GB) was used in a fully self-hosted setup, providing full control over data and model. Performance was evaluated against a reference standard created by four clinical experts and compared with a range of LLMs as used in a conventional single-prompt approach, using bootstrap resampling to estimate confidence intervals. Results: SIFTING-T-stage achieved an accuracy of 90% (95% CI: 84-95) against the reference standard. We found its performance to be comparable to even the largest state-of-the-art LLMs with reasoning capabilities and to be interchangeable with clinical experts (p < 0.001), while at the same time offering full traceability through source text references. Conclusion: SIFTING enables accurate, structured, and traceable information extraction from clinical free-text documents. It ensures data control, reproducibility, and verifiable outputs that can support clinical validation and workflow integration.