Revealing hidden details of neural networks from output alone
Cryptanalytic Extraction of Neural Networks Without Known Architecture Assumption
Cryptography and Security
Summary
Knowing the exact setup of a neural network has been assumed necessary to copy its inner workings by only observing its responses. This paper challenges that by showing how to figure out both the structure and the settings of certain neural networks just by looking at their outputs. The authors discovered clues that appear when guessing the network's size incorrectly, which help narrow down the real structure step by step. Their method works on various common neural network types and removes a key assumption in earlier approaches.
What this means in practice
- •For ai security teams: Identify potential risks in AI models by extracting both structure and parameters from their outputs alone, improving model security assessments.
- •For machine learning developers: Evaluate robustness of neural network models against extraction attacks that do not rely on known network architectures.
Authors
Yantian Shen, Yi Chen, Anyu Wang, Hongbo Yu, Xiaoyun Wang
Abstract
Cryptanalytic extraction attacks recover the parameters of a neural network given only black-box access to its raw output. However, all existing attacks rely on a fundamental assumption: the attacker knows the network architecture. For example, regarding ReLU activation-based fully connected networks, the network depth and the dimension of each hidden layer are known. In this paper, we study whether this assumption can be removed. We focus on ReLU fully connected networks and propose a guess-and-determine framework that recovers the architecture and the parameters jointly. The core of our approach is a simple but powerful observation: dimension guessing leaves architecture-sensitive traces in the parameter recovery process. We identify two such traces: (i) a \emph{zero suffix} in the merged weight vectors produced by signature recovery, whose length reveals the number of excess guesses; and (ii) an \emph{equality pattern} in the preimage-based sign recovery, which occurs only when the dimension guess is correct. These two signals give rise to two complementary recovery routes. We further propose two criteria for identifying the second-to-last layer, which is necessary for terminating the guessing process. We implement end-to-end attacks on a wide range of ReLU networks, including both expansive and non-expansive architectures. To the best of our knowledge, this is the first cryptanalytic extraction attack that removes the assumption of known network architecture.