Large language models change answers based on question wording
FramingQA: Does the Question Shape the Answer? Measuring the Compositional Framing Effect
Computation and Language
Summary
Sometimes, the way a question is asked can change the answer a large language model gives, even if the facts stay the same. The authors created a test called FramingQA to see how much these models are affected by different ways of asking questions in fields like law, medicine, and finance. They found that models might give different advice just because the question has a hidden assumption or is phrased in a certain way. This can be a problem, especially when people rely on AI for important decisions. The study shows that even when models get answers right for one version of a question, they might not be consistent if the question is asked differently.
large language modelsquestion framingframing biasmodel robustnessnatural language processingcompositional effectsbenchmarkhigh-stakes domainsmodel sensitivityrephrasing
Authors
Hazel H. Kim, Andrew M. Bean, Guilherme Affonso Ferreira de Camargo, Shanyu Chauhan, Felix Drinkall, Jade Kosché, Chenyang Ma, Glory Nwaugbala, Nabeel Seedat, Bradley Max Segal, Samuel Recht, Hinrich Schütze, Philip H. S. Torr
Abstract
We introduce FramingQA, a benchmark that measures the model sensitivity to question framing across law, medicine, finance, and robotic simulations. Large language models (LLMs) often change their responses to subtle rephrasings that align with an implied stance by users. This can leave users with advice tainted by how they happened to phrase a question rather than by the underlying facts, and the consequences are highly costly in high-stakes domains. Because in the realistic scenarios, both expert practitioners and non-expert users frequently ask LLMs questions containing incomplete or misleading assumptions, models are highly susceptible to those framings. To test this, we inject the framing bias across three nested levels: a framing-biased question phrasing (root), an injected framing-biased premise prepended to a neutral question (propositional), and a premise paired with a framing-biased question (global). Evaluating nine open models (3.8B-70B) across four families, we find that strong per-variant accuracy does not guarantee the robustness across differently phrased questions under the fixed factual information.