Language models struggle to spot scientific conflicts in complex reports
When Scientific Contradictions Are Lost in Translation
Artificial IntelligenceComputation and Language
Summary
Sometimes two scientific studies seem to disagree but are actually describing different things. The authors tested language models to see if they could figure out when findings contradict each other or not. They found that when the scientific statements are written in complex, natural language, models often prefer explanations that match what they expect biologically rather than what the actual supporting data shows. This means language models depend on both formal logic and understanding of scientific context to verify results correctly.
What this means in practice
- •For biomedical data teams: Improve automated checks for conflicting scientific reports by combining formal logic with biological context understanding.
- •For ai developers: Enhance language model training and prompts to better distinguish compatible versus contradictory scientific results in natural language.
Authors
Tal Zeevi, Trey W. Jensen, Maxwell Strome
Abstract
Two scientific findings can disagree without contradicting each other. Determining whether they conflict requires knowing whether they describe comparable measurements. We study how language models behave at this decision point. In a controlled task, we generate an unsatisfiable XOR constraint system and translate its constraints into scientific reports from different laboratories. One assignment satisfies more constraints, while another satisfies fewer but better matches expected biology. This creates a simple dilemma: does the model choose the assignment that best fits the constraints, or the one that better matches biological expectations? When the constraints are stated directly, GPT-5.6 Sol and Claude Opus 5 recover the best-supported assignment in 90% and 96% of cases, respectively. In scientific prose, however, the models behave differently. Claude Opus 5 often prefers the biologically expected assignment. Removing that biological preference increases recovery of the better-supported assignment from 27% to 79% (p<.001); recovery reaches 92% when the same Biology-favored record is accompanied by a formalization request and an explicit paired-design cue (p<.001). GPT-5.6 Sol is less sensitive, with neither corresponding change reaching statistical significance. These results suggest that reliable scientific verification depends not only on formal reasoning, but also on how models decide which findings should be compared and what relations they imply.