Papers for

software qa engineers

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

Decision models judge answers differently based on request formatting

Adversarial Cues in Decision Models Used as Judges: The Role of Request Presentation

Abstract: An answer judge instructed to grade the final commitment should reject an explicitly wrong final value even when an earlier value matches the reference. We show that adding one colon to a candidate can violate this requirement depending on the presentation of the structured judging request. Numeric references certify the error, and paired interventions distinguish the candidate edit from the integration's presentation choices. On 200 previously unused DROP and GSM8K source clusters, the edit increased Jev's false acceptance from 1.0% to 26.0% with three output labels and from 3.0% to 26.5% with the published four-label grading instruction under sorted JSON keys. Both candidate variants were rejected under insertion presentation. These interactions passed the prespecified statistical correction even though Jev met the control thresholds under both grading configurations and presentations. Most excess acceptances occurred among candidates assigned larger numerical errors. GPT-6 Sol produced no observed cue-condition false acceptances, with missing responses unresolved. The result shows that basic judging competence can coexist with sharply different vulnerability to a fixed candidate edit across logically equivalent request presentations. The reference-aware grammar and compound ordering change limit the finding's operational scope and leave its internal cause unmeasured.

Thu 8 OctComputation and LanguageCryptography and Security
The gist
The paper shows that the way a question is presented to an AI judge can influence whether it correctly rejects a wrong answer. The researchers found that adding a simple punctuation mark like a colon can cause the system to wrongly accept an incorrect final answer depending on how the grading request is structured. Tests with different presentations revealed these vulnerabilities even though the AI appeared competent otherwise. Another model, GPT-6 Sol, did not show these errors but sometimes did not answer. This means that small changes in input format can greatly affect AI judgment without changing the logic.
Open → 2610.11436v1