Local large language model systems can fail to keep prompts private
The Illusion of Local Privacy: Confidentiality Boundary Failures in Consumer LLM Serving Systems
Cryptography and Security
Summary
Keeping large language models on your own device is often thought to keep your messages private. This paper shows that just running the model locally is not enough to guarantee privacy, because the software handling the messages can leak information at several steps. The authors test different parts of popular local model systems and find ways that your input can be recovered or exposed by memory leftovers, saving functions, or bugs in the software interface. They conclude that local LLM setups need clearer rules for how your messages are stored and isolated.
What this means in practice
- •For local ai software developers: Improve privacy features by securing memory handling and storage of user prompts in local LLM products to prevent unintended data exposure.
- •For cloud service security teams: Design better tenant isolation and authorization checks for multi-user LLM serving platforms by learning from local system vulnerabilities.
Authors
Youssef Hamdi Zafan Ibrahim, Muhammad Ikram, Mohammed Khalaf Salama
Abstract
Running large language models (LLMs) locally is often considered more private than cloud-hosted inference because user prompts remain on the device. We ask whether keeping inference local is, by itself, sufficient to keep those prompts confidential. Our results show that it is not: prompt confidentiality also depends on how the surrounding serving software handles prompt data before, during, and after inference. We examine four boundaries at which prompt confidentiality can fail in consumer local-LLM serving systems: model loading, runtime memory, wrapper-level persistence, and the serving interface. To study these boundaries, we develop LLAnalyzer, a measurement framework that tests each boundary separately and traces observed failures to the responsible software component. Applying LLAnalyzer to four open-weight model families and two consumer deployment platforms, we find markedly different behaviour across boundaries. In a 24-hour AFL++ campaign with more than 12 million executions, we observe no parser crashes or successful malformed GGUF loads within the explored state space. Runtime memory tells a different story: we recover prompts after inference because multiple plaintext representations survive in allocator-managed memory, and sanitisation reduces this residue without eliminating it. We also find that consumer wrappers can extend prompt lifetime through plaintext persistence. At the serving boundary, we uncover a previously undocumented authorization flaw in llama.cpp that allows one authenticated client to restore another tenant's saved conversation state; the attack succeeds in 200/200 controlled trials. Separately, shared prompt-prefix caching exposes a remote timing oracle that remains distinguishable under WAN conditions. We argue that local LLM systems need explicit guarantees for prompt lifetime, persistent storage, and tenant isolation.