Ventor-QTest: Threat-Model-Driven Verification of Vendor-Hosted LLM APIs
2026-08-17 • Cryptography and Security
Cryptography and SecurityArtificial Intelligence
AI summaryⓘ
The authors address the challenge of checking the quality of third-party APIs that run open-weight large language models. They introduce Ventor-QTest, a method that tests these models without needing internal details, by sending repeated requests and analyzing the variation in their outputs. They measure two types of fidelity loss—average (AFL) and extreme (EFL)—to understand how consistent and reliable the models are, especially in long, complex tasks. Their findings suggest that extreme fidelity loss is linked to performance drops in tasks requiring longer reasoning. The authors recommend using both AFL and EFL together when auditing these language model APIs.
large language modelsopen-weight modelsblack-box auditingAPI qualityfidelity losscoarsened-KL divergencestochastic processlong-horizon tasksTerminal-Benchsurprisal statistic
Authors
Xiangfan Wu, Zonghao Ying, Huiyu Wu, Xing Zheng, Huangsheng Cheng, Xiaorong Shi, Jing Guo
Abstract
As large language models become increasingly widespread, third-party providers that deploy open-weight models have become an important part of the ecosystem. Auditing the quality of their inference APIs is therefore an open problem. We formalize hosted model routing as a stochastic process and propose \mbox{\textbf{Ventor-QTest}}, a composite black-box audit that requires no probability information from the target API. Its repeated-request component sends each frozen constrained context to the target multiple times, reconstructs a categorical output distribution from the returned text counts, and reports \emph{average fidelity loss} (AFL) as a null-bias-corrected, within-window mean coarsened-KL statistic. Its long-sequence component uses independent runs to report \emph{extreme fidelity loss} (EFL) through the empirical upper tail of a run-level reference-centered-surprisal statistic. Across three logprob-capable route conditions, AFL shows strong linear descriptive agreement with a logprob-derived coarsened-KL comparator. Across seven route snapshots, 20-run sequence probes reveal route-specific EFL variation. AFL and EFL have little detectable route-level association with GPQA-Diamond accuracy. In contrast, pronounced EFL coincides with a decline in Terminal-Bench pass rate as task exposure increases. This pattern may arise because correctness in long-horizon tasks is more sensitive to extreme fidelity loss. These results motivate reporting AFL and EFL jointly, particularly when auditing long-horizon agentic tasks. The open-source implementation is available at https://github.com/Tencent/AI-Infra-Guard/tree/main/services/api_checker/ventor_qtest.