Summary
Some computer models decide things based on questions and give answers with confidence scores, used without humans reading them. The authors found it hard to test how reliable these models are because they don't always give the same answer to the same question and often use their own outputs as labels. They made a new way to test these models by comparing model answers before and after small changes to the input. Their tests show certain small edits can change many model decisions noticeably, especially when adding unverified information. This means systems using these models should treat input data carefully, as some changes can fool the model.
What this means in practice
- •For software engineers: Test the robustness of decision-making AI models against subtle input changes using JevAdvBench's benchmark and attack suite.
- •For security teams: Identify potential vulnerabilities in AI decision models by simulating adversarial inputs that cause incorrect or uncertain decisions.
Authors
Jianyi Hu, Hangtao Zhang, Yi Liu, Yeqi Zeng, Li Zeng, Xianlong Wang, Rui Wang, Leo Yu Zhang
Abstract
Models trained with reinforcement learning for calibrated decisions (RLCD), such as Jev, answer a typed question about an input, the state, with a probability, a choice, or a score, and software acts on the answer without a person reading it. Their robustness has not been measured: adversarial benchmarks score what a model generates or executes, whereas a typed model generates nothing and returns a well-formed answer even when manipulated. Measurement is also hard, because identical requests can return different answers, most available labels come from the model itself, and the API preprocesses each request out of view. Our key idea is to score each attacked decision against the model's own clean decision rather than against labels, and to read it against the change caused by an identical re-run. Building on this, we introduce JevAdvBench, to our knowledge the first adversarial benchmark for RLCD models, with 812 typed questions over 66 scenarios, and a black-box attack suite of 9,744 single-edit variants that each edit one part of a request, with billed input tokens confirming that the edit reached the model. On jev-1.13.0, rewording stays within 1.2 percentage points of the re-run baseline, and fields outside the schema never reach the model. In contrast, one unverified opinion appended to the state flips 12.1% of decisions, statistically tied with the strongest injected command (10.1%), and pushes 38% of confident answers below the 0.8 confidence threshold that routes them to human review. Applications built on RLCD models should therefore treat the state as untrusted, argued input. Project website: https://JevAdvBench.github.io/JevAdvBench/