Summary
This paper looks into how to reverse-engineer the inner workings of certain types of neural networks by combining cryptographic techniques with learning from the network’s predicted labels. The authors adapt methods to handle different activation functions and provide mathematical guarantees about when and how well parts of the network can be recovered. Their experiments show that they can recover many neurons under certain conditions, even without access to the internal weights, by only using predictions and some privileged information. However, they also explain why simply agreeing on predictions doesn’t mean the network’s parameters are exactly known.
What this means in practice
- •For security analysts: Assess risks of exposing model predictions by reconstructing internal network parameters through label queries and partial gradient access.
- •For machine learning engineers: Improve model distillation techniques by combining label-based learning with direction and sign extraction methods to better approximate target networks.
Authors
Biprarshi Biswas, Arya Anshuman Ojha, Yubaraj Das, Devansh Khosla, Kamran Rais, Saankhya Samanta, Subhamoy Maitra, Pantelimon Stanica
Abstract
We study fully connected neural-network reconstruction by combining cryptanalytic weight-direction extraction with learning from predicted labels. We adapt a ReLU extraction implementation to Leaky ReLU, compare sign-search methods, and estimate biases and output layers. We prove conditional bounds for direction recovery with approximate prefixes, preservation of local distance votes, prediction changes from aligned parameter errors, and sign selection with independent scoring data. We also specify exact row restoration and distinguish its guarantee from that of gradient masking. The experiments evaluate assisted extraction, with privileged access to gradients, hidden activations, and victim parameters, followed by label-based reconstruction. Across eight search seeds on the largest synthetic architecture, Leaky ReLU recovers 51--57 neurons in the deepest hidden layer, while ReLU recovers none. This advantage does not hold uniformly across architectures and layers. Removing victim-weight access from the sign walk gives less accurate signs, despite high final prediction agreement. Across 40 seeded runs on the two smallest synthetic victims, agreement on 5,000 inputs excluded from fitting and selection ranges from $0.9890$ to $0.9992$. On two CIFAR-10 victims, the reconstructed models achieve $56.40\%$ and $58.29\%$ agreement and exceed the distillation baseline on three functional metrics. We report label requests separately from privileged access and explain why prediction agreement does not identify network parameters.