Hybrid encryption speeds up private large language model decoding
ROSETTA: Efficient and Accurate Privacy-Preserving LLM Decoding via Hybrid CKKS/TFHE Evaluation
Cryptography and Security
Summary
Large language models can create text but using them privately is slow because of how encrypted calculations work, especially for complicated steps. The authors found that by mixing two types of encryption methods, they could handle different parts of the problem more efficiently. They built a system called ROSETTA that chooses the best method for each step, making the process much faster while keeping data secure. Their tests show ROSETTA speeds up some key operations by almost five times and overall decoding by up to twice as fast compared to previous tools.
What this means in practice
- •For cloud service providers: Securely run language model queries on encrypted user data with improved speed and privacy.$Commercial implications: Enables faster encrypted AI cloud services for clients requiring privacy protections.
- •For enterprise security teams: Deploy privacy-preserving language model inference internally to protect sensitive data during automated text generation tasks.
Authors
Jiangrui Yu, Baosheng Zhang, Liang Kong, Lin Ding, Yi Chen, Ye Yu, Mingzhe Zhang, Meng Li
Abstract
Generative large language models (LLMs) have achieved state-of-the-art performance on many real-world tasks such as code generation and question answering. These models predominantly rely on an autoregressive decoding strategy that generates output tokens sequentially. However, their pervasive deployment raises serious privacy concerns, motivating private inference frameworks based on fully homomorphic encryption (FHE). A major limitation of existing FHE frameworks is their inefficiency in evaluating nonlinear operations, which incur substantial overhead and dominate the decode stage. In this paper, we propose ROSETTA, a hybrid CKKS/TFHE framework that overcomes this limitation. We first observe that nonlinear operations in the decode stage exhibit heterogeneous workload patterns, which can be handled effectively via a hybrid approach. We then realize this with two key contributions: 1) an adaptive segmented lookup-table protocol based on TFHE that enables efficient and accurate evaluation of nonlinear operations; and 2) a scheme-aware operator-selection framework that automatically assigns each nonlinear operator to CKKS or TFHE to minimize end-to-end decoding latency. We demonstrate that ROSETTA achieves up to $4.8\times$ Softmax speedup and $1.5$--$2.1\times$ end-to-end speedup over the SOTA framework CacheMir.