Permutation shuffling fails to protect model secrecy in encrypted AI inference
Shuffling is Not Enough: Breaking Permutation-Based Model Confidentiality in Hybrid FHE Inference
Cryptography and Security
Summary
This paper shows that simply mixing up (shuffling) the outputs of an encrypted AI model is not enough to keep its details secret during private computations. The authors demonstrate that an attacker can recover key parts of a model’s inner workings with only a small number of cleverly chosen queries. They prove that the usual noise added to protect privacy cannot fully hide the model without ruining accuracy. Their methods work on popular neural networks, revealing model details perfectly in tests.
What this means in practice
- •For cloud ai service providers: Detect and prevent unauthorized extraction of AI model details in encrypted inference services using the paper’s attack strategy.
- •For security engineers: Evaluate and improve confidentiality guarantees in hybrid FHE deployment by understanding how permutation-based protections can be bypassed.
Authors
Jiseung Kim, Hyung Tae Lee
Abstract
Hybrid fully homomorphic encryption~(FHE) inference improves the practicality of private inference by letting the server evaluate linear layers homomorphically while the client decrypts and applies nonlinearities. Recent schemes attempt to protect model confidentiality by returning noisy, output-permuted responses and appealing to shuffle-model differential privacy~(DP). We show that this protection fails in the correctness regime required by hybrid FHE systems. For a $d$-input linear layer, $d+1$ admissible queries suffice for exact recovery of a permutation-invariant layer summary, hence for perfect model distinguishability. We further show that input DP is orthogonal to model confidentiality and that the local-DP premise required for shuffle amplification cannot hold under correctness-bounded noise. We recover all linear layers of a \safhire{}-style ResNet-20 end-to-end from TFHE transcripts with zero error, using $d+1$ queries per layer for a total of $5{,}712$ direct queries. Under the same query model, we also confirm exact per-layer recovery on pretrained ImageNet-scale CNNs and ViT-B/16. The leaked spectra enable fingerprinting, lineage attribution, and improved logit-based extraction, while suppressing them destroys inference utility.