Attack resistant fingerprinting methods protect multimodal large language models
Fingerprinting Multimodal Large Language Models
Cryptography and SecurityArtificial Intelligence
Summary
Multimodal large language models (MLLMs) that understand images and text can be stolen or copied without permission. The authors found ways to create unique 'fingerprints' from how these models attend to information, which helps prove ownership if someone uses the model unlawfully. They also developed a way to detect if a model has been copied (distilled) by analyzing its outputs. Their methods were tested on many models and can resist attempts to hide or alter the fingerprints.
What this means in practice
- •For ai security teams: Identify unauthorized copies or usage of proprietary multimodal language models through fingerprinting methods robust to common modifications.
- •For software compliance auditors: Detect potential illicit distillation or derived versions of multimodal models by statistically testing model outputs.
Authors
Chao Huang, Meng Tong, Kejiang Chen
Abstract
While multimodal large language models (MLLMs) enable a wide range of image-text reasoning tasks, recent incidents indicate that they are vulnerable to illicit deployment and unauthorized distillation. Existing solutions for model provenance are typically confounded by shared language backbones in MLLMs and struggle to detect violations of distillation. To bridge this gap and safeguard model ownership, we present the first study on multimodal model fingerprinting. Inspired by recent findings that self-attention acts as a low-pass filter and that its low-frequency components are informative, we develop AttnPrint for white-box provenance. Specifically, we extract cross-modal attention distributions and isolate their low-frequency components to serve as model fingerprints. To facilitate black-box auditing, we further introduce DistillTrace, which employs hypothesis testing of MLLM outputs to identify potential model infringement. We conduct extensive experiments on 154 model instances across 19 multimodal architectures. Notably, AttnPrint achieves strong derivative-model detection performance while remaining robust to five downstream modification techniques. DistillTrace also provides evidence of distillation relationships under three parameter-independent techniques.