Recognition-Refusal Misalignment in LLMs: Why Models Answer Structurally Unanswerable Questions
Computation and Language
Summary
The authors studied why large language models sometimes try to answer questions that can't be answered, like impossible math problems or incorrect code, instead of saying they can't answer. They found that these models do recognize when a question is impossible, shown by a specific pattern in the model's internal state. However, this recognition is separate from the part of the model that decides to refuse harmful or unsafe questions. This means the problem is not that the model doesn't understand the question is impossible, but that it doesn't properly connect that understanding to choosing not to answer. So, the error is about how the model routes information, not about what it knows.
Authors
Yucheng Du, Xiyang Hu
Abstract
Large language models often answer structurally unanswerable questions, such as computing cot(-540°) or evaluating (1).startswith("1"), instead of abstaining. We ask whether this failure reflects missing recognition or failed routing from recognition to abstention. Across instruction-tuned models from 1.7B to 70B parameters, a single linear direction in the hidden state separates answerable from structurally impossible math and code prompts, showing that models represent impossibility before generation. Yet this recognition direction is nearly orthogonal to the canonical safety-refusal direction that mediates trained harmful-content refusal. An in-domain behavior-defined invalidity-aware direction is closer to recognition, but only partially aligned with it, and remains near-orthogonal to safety refusal. Generation-time steering along the recognition direction changes invalidity-aware behavior bidirectionally and dose-responsively on structural math and code cells, while random directions do not. Base/instruct comparisons further show that the low-cosine geometry is already present at the pretraining endpoint. The confident-on-impossible failure is therefore better explained as a routing failure than as an encoding failure: the model has a usable "no admissible answer" signal, but the safety-refusal pathway is not aligned to use it.