ReLU networks reveal more by analyzing outputs as vectors not scalars
Beyond Scalar Probes: Exploiting Vector-Valued Outputs in ReLU Networks For Signature Extraction
Cryptography and Security
Summary
This paper looks at ways to understand neural networks that use ReLU functions by not just looking at one output at a time but the entire output vector. The authors found a neat mathematical way to describe how the network changes between different small regions, which lets them see more information than before. Their approach works better than older methods when computers store numbers with different levels of precision. They also tested it in situations where numbers are stored less precisely, similar to what many real-world systems use.
What this means in practice
- •For machine learning engineers: Improve extraction and debugging of ReLU-based neural networks by capturing detailed behavior across output vectors, especially under different numerical precisions.
- •For hardware designers: Optimize neural network accelerators by better understanding vector output structures in low precision environments common in edge devices.
Authors
Gorka Abad, Claude Carlet, Ermes Franch, Stjepan Picek, Vincent Rijmen
Abstract
We revisit cryptanalytic extraction of ReLU networks from a geometric and algebraic perspective. Rather than restricting attention to a single output component, we study the full vector-valued behavior across adjacent linear regions. This leads to a rank-one characterization of Jacobian differences that recovers the usual row-signature information while also revealing complementary column-side information. Our experiments show how this additional structure can be used in ex- traction and improves numerical estimation under different numerical- precision regimes (float64, float32 and float16). We extend the analysis beyond the high-precision and output-rounding settings commonly con- sidered in the literature towards the low-precision settings encountered in many practical settings.