Transparency in large language models can reduce security and trust
Can Open Be Too Open? On the Security and Trustworthiness of Fully Transparent LLMs
Cryptography and Security
Summary
People often believe that making large language models (LLMs) fully open and transparent—sharing their code, data, and training steps—will make them safer and more trustworthy. The authors argue that this isn't always true because it's very hard for users to check if the open parts really match each other, so trust in the developers is still needed. Also, being fully open can help attackers exploit the models more easily, and current ways to fix these security issues aren't strong or simple. The authors suggest that as technology changes, these ideas might need to be reviewed in the future.
What this means in practice
- •For ai platform operators: Decide when to openly release LLM components by weighing transparency benefits against increased attack risks under current security limits.
- •For cybersecurity teams: Develop or improve defense methods knowing that full openness of LLMs can raise vulnerability exposure despite its trust benefits.
A position paper. It proposes an approach and reports no results.
Authors
Sheila Zingg, Florian Tramèr, Martin Vechev, Srdjan Čapkun
Abstract
There are different levels of transparency for LLMs: open weights, open data, and/or open pipeline. Recently, more interest has emerged in fully transparent LLMs, i.e., LLMs with fully open weights, data, and training pipeline. Drawing analogies to other fields such as software security, it is often believed that increasing transparency inherently increases the security and trustworthiness of the LLM. However, we argue that such claims need to be critically re-evaluated in the context of LLMs. In this paper, we analyze the impact of transparency on LLM security and trustworthiness by reviewing the relevant literature. Our core findings are that, due to the limited reproducibility of LLM pipelines, in practice, users cannot verify the correspondence between open weights and the associated open data and pipeline. Thus, in order to trust a model, users still need to trust that developers acted honestly, reducing the benefits over models that provide less "transparency". Moreover, we find that even when LLMs are fully transparent and trusted, this openness can favor adversaries. Open weights in particular, but also open data and pipelines, can increase the success rate of attacks. Although the same mechanisms can be used to improve the model and protect it against attacks, the current lack of robust lightweight mechanisms to patch LLM vulnerabilities limits the benefits of openness at this time. We note that our analysis is based on current LLM capabilities and analysis tools, and that future developments may require a re-evaluation of these points.