Summary
When companies try to protect their AI models from being copied by many different user accounts working together, they often look at the questions these accounts ask. The authors studied whether just looking at these queries is enough to spot copying campaigns. They found that such detection is only possible if the group copying looks more unusual than any normal group of users. Small differences only affect how quickly detection happens, not whether it happens. They also show that some clever attacks can avoid detection but result in weaker copies of the model. Finally, they note important limits, like slow copying that confuses detectors or copying that blends in with real user requests.
What this means in practice
- •For ai security teams: Detect coordinated model extraction campaigns by comparing query patterns against normal user behavior to identify unusually synchronized queries.
- •For online service operators: Improve monitoring tools to detect and mitigate model theft attempts by analyzing query timing and topical similarity among multiple accounts.
Abstract
Model providers now face distillation campaigns run through thousands of accounts, each of which looks like an ordinary user. When network and payment metadata cannot link those accounts, the only evidence left is what they ask. We study what a provider can and cannot learn from those queries alone, using one million real chat turns for development and 17 later weeks that no design choice touched for a single frozen evaluation. The answer is set less by anything specific to extraction than by benign coordination: a campaign is visible only if it is more distinctive than the most distinctive group of ordinary users in the same week. We make this precise, show that a small but positive difference between campaign and users buys a detection time rather than detectability, and identify the campaign that is hardest to detect. Empirically, a calibrated monitor catches naive distributed extraction that per-account monitors miss (0.86-1.00 of weeks), and one variant - selecting accounts that move together after discounting how topical each one is - still catches campaigns padded with natural text (0.50-0.96 of weeks) where the other two spectral detectors catch none. Distilling each campaign into a small student shows that, in our testbed, campaigns that evaded every detector kept at most 42% of the capability of a detectable one. The limits are equally concrete: a campaign that ramps up slowly poisons every detector's calibration, benign evaluation pipelines look exactly like distillation, we found no deployable detector at colluder shares of 2.5% or below, and campaigns that relay real users' requests may be invisible to any query-only monitor.