Summary
Running transformers securely to protect private information usually takes a lot of extra computing power, especially for complicated math steps like Softmax and GeLU. The authors show that existing ways to compress these models speed things up but at the cost of accuracy when combining different methods. They created DegreeSpar, a new method that simplifies these complex parts by focusing on the math degree involved, allowing both token and model size reduction in one step while keeping accuracy high. DegreeSpar trains models to accept simpler math formulas for faster operation and achieves notable speed improvements across different tasks and transformer sizes.
What this means in practice
- •For machine learning engineers: Improve the speed of privacy-protected transformer models in natural language and vision tasks while maintaining accuracy.
- •For cloud security teams: Deploy faster secure transformer inference systems to handle sensitive data with reduced computational overhead and cost.
- •For data privacy software developers: Build secure AI applications that run encrypted transformer models efficiently using structured polynomial degree sparsity.$Commercial implications: Enables creation of commercial privacy-focused AI services offering faster and accurate encrypted model predictions.
Authors
Yifei Cai, Zhuoran Li, Xiaozuo Shen, Hongyi Wu, Chunsheng Xin
Abstract
Secure Transformer inference protects sensitive inputs but incurs substantial cryptographic overhead, with nonlinear operations such as Softmax and GeLU becoming major bottlenecks. Existing compression methods reduce nonlinear complexity, sequence-dependent computation, or model structure through separately defined compression variables. Under aggressive compression, however, these independently optimized perturbations can accumulate: at a matched compression level, stacking representative approximation, token-pruning, and model-pruning methods reduces ViT-S accuracy from 80.20% to 76.41%. We introduce DegreeSpar, which formulates secure Transformer compression as structured sparsification over nonlinear polynomial degrees. Polynomial degree directly controls the cost of secure nonlinear evaluation, while computation-aligned zero-degree structures expose token-level and model-dimension computation as removable within the same optimization space. DegreeSpar further incorporates approximation-aware training for low-degree Softmax and GeLU, enabling aggressive degree reduction and creating the optimization headroom required for structured computation removal. Across vision and language Transformers, DegreeSpar consistently improves the accuracy-latency trade-off across model scales, tasks, and sequence lengths, achieving speedups from 2.29x to 6.63x over the corresponding baselines. Under the same network setting, DegreeSpar achieves 92.68% accuracy on BERT/SST-2 in 110.55 s, compared with 92.66% in 167.26 s for CipherPrune, the closest prior hybrid secure-inference approach. These results establish structured polynomial degree as an effective shared optimization space for secure Transformer compression.