Language models identified by unique one-word answers to questions
Forty Questions: Identifying a Language Model from One-Word Answers
Computation and Language
Summary
It is possible to tell which language model answered a question by looking at just one-word responses to simple everyday questions. The authors show that by collecting patterns of these answers from many models, they can guess the right model over 90% of the time. They tested this on a large number of models through APIs and found some mismatches and hidden details like extra billing tokens added by certain services. This method is low-cost and relies on answers from common questions, making it practical to identify models behind APIs.
What this means in practice
- •For api developers: Detect which language model is serving an API based on its one-word answers to standard questions.
- •For security teams: Identify and verify language models in use to audit unauthorized or mismatched API deployments by comparing answer profiles.
- •For customer support platforms: Confirm the underlying language model of third-party chat services to ensure compliance and billing transparency.$Commercial implications: Enables verified model identification to provide trustworthy billing and service claims for platform operators.
Authors
Tapan Parikh
Abstract
A model behind an API can be identified from its one-word answers to everyday questions such as "Name a sound" or "Name a poet". A model's profile records how often it gives each answer to each question. We attribute new answers to the model most likely to have given them. On a panel of 94 models, 40 answers identify the exact model 92.5% of the time, against 1.1% by chance. Profiles built through OpenRouter identified all ten models served by DeepInfra under blinded labels. A system prompt lowers accuracy by 6 to 16 points. We applied the method to 96 endpoints of 13 open models on OpenRouter, and 92 identify as the model they claim. Of the other four, one matches the claimed model with reasoning turned off. One serves Gemma 4 under the name Llama 4 Maverick: it names itself Gemma 4, and it is billed Gemma 4's token counts on every message. Two remain unresolved. Billing records also show context added to first-party models. Grok 4.7 is billed 1,247 input tokens for a one-line question that costs GPT-4o only 12 tokens, and five models state the current date without being billed for it. The questions come from a public census of model defaults, and the method costs cents per endpoint.