Large language models detect misleading outcome reports in clinical trials
Detecting Spin in Clinical Trials with Large Language Models
Computation and Language
Summary
Sometimes, reports of medical studies make results look better or different than they really are, which can mislead doctors and patients. The authors created a way to use large language models (LLMs) to spot when study results have been switched or changed unfairly. They tested their method on thousands of examples and found it was better than basic text-matching tools, though not quite as good as some specialized AI models. They also used the LLMs to explain why they thought a report was misleading.
What this means in practice
- •For medical journal editors: Automatically flag reports with switched or misleading clinical trial outcomes to improve review quality and decision making.
- •For regulatory compliance teams: Use the method to detect distorted outcome reporting in submitted trial documents for faster integrity checks.
Authors
Tjaš Ajdovec, Marko Robnik-Šikonja, Simon Šuster
Abstract
Spin in clinical trials includes reporting practices that distort the presentation of results. This is particularly critical in medicine, where spin is present in more than 50% of randomized controlled trials that fail to reach statistical significance. The comparison of primary and reported outcomes is crucial for detecting several types of spin, including outcome switching. We used 300 pairs of outcomes labeled with semantic similarity to develop a system for automatic detection of outcome switching. We evaluated baseline text similarity models and open-source LLMs using generated similarity scores and the Youden index to determine the classification threshold. The proposed approach involves prompt engineering, classification based on token probabilities, and majority voting for the final decision. The results on the test set of 2,496 examples with an F1 score of 0.78 and an accuracy of 0.90 outperform baseline text similarity models but trail behind fine-tuned versions of BERT. We used LLMs to generate natural language explanations for the classified instances and manually assessed their quality.