Arctic science questions help test AI model answers and abstentions
Arctic Questions, Missing Answers: A Dataset and Benchmark for LLM Abstention in Arctic Science
Computation and LanguageArtificial Intelligence
Summary
Sometimes big language AIs are asked scientific questions where none of the answer choices are right, but they don’t always say “I don’t know.” The authors made a special set of 194 real questions about Arctic science to see if these AIs would choose to abstain when answers are wrong or missing. They tested 8 AI models and found big differences in when and how often the models chose not to answer. This helps understand how to measure if AI really knows when it should say it can’t answer.
What this means in practice
- •For ai system developers: Test and improve AI models’ ability to withhold answers when no valid choice exists, improving scientific question answering reliability.
- •For environmental data teams: Use ArcticQA to verify that AI tools analyzing Arctic research avoid incorrect conclusions when scientific evidence is missing or contradictory.
Authors
Benjamin Wilcox, Dawei Gao, Pradeeban Kathiravelu, Douglas Causey, Kewei Sha, Yunhe Feng
Abstract
Large language models (LLMs) should abstain from scientific multiple-choice questions when no option is valid, but frequent abstention alone does not demonstrate sensitivity to answer availability. We introduce ArcticQA, a dataset of 194 questions derived from primary Arctic research, with automated checks of answer support and distractor contradiction against source evidence. We further develop ArcticAbstain, a paired benchmark comparing answer-present and answer-absent conditions, with the correct answer replaced by a distractor in the latter and an explicit abstention option in both. We evaluate eight models from the Gemini, Claude, and ChatGPT families at high reasoning effort, with three trials per condition, yielding 9,312 recorded responses. Answer-present abstention rates range from 0.0% to 63.0%, whereas replacing the correct answer increases abstention by 5.05 percentage points on average. These findings highlight substantial baseline differences and the need to evaluate abstention frequency and responsiveness jointly. The dataset and benchmark are available at https://github.com/BenWilcox8/arctic-qa.