Privacy enhanced inference on large AI models using transformed inputs
Privacy-preserving inference on hosted models: a learned input-embedding transform for frozen Qwen3.8-27B
Artificial Intelligence
Summary
When people use big language models hosted on other computers, their typed questions are visible to those hosts. The authors study a way to keep the original questions private by turning the words into special number patterns locally before sending them. These transformed patterns let the big model still understand and answer well, with only a small drop in accuracy. This method means the host never sees the real text, improving user privacy.
What this means in practice
- •For cloud ai service providers: Integrate a local vector transformation step to protect client text data while running hosted large language models without exposing prompt content.
- •For enterprise ai developers: Build client-side tools that convert inputs into transformed embeddings, enabling privacy-preserving queries on large frozen language models in sensitive settings.
Authors
Mahendra Roopa, Srikanth Roopa
Abstract
Every prompt sent to a hosted language model is readable by whoever runs it. We study whether the prompt can remain on the client while the host receives transformed vectors that stand in for it. A language model converts text to vectors exactly once, with a public lookup table; we perform that step locally and pass the vectors through a small trained network (55.6M parameters) before they leave the perimeter. The host runs a frozen 27B model (Qwen3.8-27B) on the transformed vectors and never receives text or token ids. We train the transform against the frozen model as a fixed judge, combining a forward-KL utility term with an auxiliary term that pushes each vector away from its original direction. On 1,490 MMLU questions held out from training, the transform costs 1.5 points of accuracy (80.7% vs. 82.3%), breaking 68 of the 1,226 questions the unmodified model answers correctly (5.5%), while 30.1% of content tokens no longer decode to their original token under a nearest-neighbour lookup.