Papers for

security teams in ai companies

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

Multi-agent systems are only as safe as their weakest model

Only as Safe as the Weakest Agent: A Break That Appears Only When the System Is Multi-Agent

Abstract: Major industry multi-agent systems like Codex and Claude Code can be broken by attacks that have long been unsuccessful against modern frontier models. AI Safety as a discipline has largely focused on finding the bounds and vulnerabilities of AI models used in isolation. Yet as humans increasingly deploy these models as agents and multi-agent systems, the same attacks that had been thought to be patched when using a model in isolation can break these same models under specific circumstances. We show that these system breaks are not purely theoretical, but can break Claude Code, Codex, and a reference harness, all unmodified and used with their own sub-agent mechanisms. We show the attack by asking agents to answer questions from a store of files on disk in which we have planted a forged record. Searching the store themselves, the strong model orchestrators commit almost zero harmful revisions in over 1,000 trials. However, if that same orchestrator model delegates to a cheaper sub-agent, which is what these harnesses are built to do, the same models on the same facts fail on 20/180, 44/180, 75/180 and 553/900. Thus, even when the system is driven by a strong model, the system's safety lands near the weak reader's, not the orchestrator's. Simulations over 18 models show the same effect in the delivery channel where five aggregators, who on their own were almost never unsafe, repeat a corrupted sub-agent's error in 751/757 trials. The safety of real-world agentic systems that use strong orchestrator agents and weaker sub-agents, and multi-agent systems in general, is defined not by the safety of the strong agent but by the safety of the weakest agent.

Thu 12 NovMultiagent SystemsArtificial Intelligence
The gist
Many AI systems use multiple models working together, where a strong main model delegates work to smaller, weaker helper models. The authors show that attacks which fail on strong single models can succeed when these models work in teams. Even if the main model is very safe, the system’s safety is limited by the weakest helper model it uses. This means that multi-agent AI can be vulnerable in ways not seen when models work alone.
Open → 2611.17520v1

Adapters can hijack decisions only after model quantization deployment

Hijacking a Trusted LLM in Real-World Quantized Deployments Only Requires a Disguised Adapter

Abstract: Third-party low-rank adaptation (LoRA) adapters are tailored to perform decision tasks such as content moderation and function selection. However, in the real world, quantized deployments carry a potential threat in that an adapter can pass an audit with the unquantized base, yet hijack decisions when the same base is quantized for deployment. Demonstrating this threat requires an adapter that preserves benign behavior before quantization, but hijacks decisions afterward. Further, this must hold without modifying the trusted base or knowing the victim's quantization configuration. We therefore present QHijack-LoRA, an adapter-only attack that uses the base model's natural quantization residual to activate decision hijacking. The attacker modifies neither the trusted base model nor the victim's quantization configuration. With QHijack-LoRA, a four-objective training procedure separates full-precision dormancy from quantized target activation, while preserving clean-input behavior. The adapter is trained across several different quantization methods, while adaptive reweighting and residual-direction augmentation extend the attack to unseen configurations. Our results show that auditing performed exclusively under full-precision conditions does not provide assurance of decision integrity in quantized deployment environments. Thus, we also propose a cross-precision paired audit that checks the candidate adapter in its intended deployment configuration. This work shows the critical need to audit third-party adapters in any potential quantized deployment.

Wed 11 NovCryptography and Security
The gist
Large language models often use small add-ons called adapters to change behavior for tasks like content moderation. This paper shows that an adapter can look harmless when checked with the full-precision original model but behave maliciously when the model is made smaller and faster through a step called quantization. The authors created an attack that works without changing the model or knowing exactly how it was shrunk, exploiting the differences introduced by quantization to hijack decisions. They also highlight the need for checking adapters both before and after quantization to prevent such hidden risks.
Open → 2611.14978v1

FocusBranch improves neural network robustness checks for sparse changes

FocusBranch: Combinatorial Branch-and-Bound for $\ell_0$ Neural Network Robustness Verification

Abstract: Verifying the local robustness of neural networks under $\ell_0$ perturbations is inherently combinatorial, as it must reason over all sparse subsets of $ε$ perturbed pixels. This paper presents FocusBranch, a combinatorial branch-and-bound framework for $\ell_0$ robustness verification. The key insight is to make the $\ell_0$-ball decomposable and branchable using a node abstraction, which consists of a focus pixel set $\mathcal{S}$ and a level $l$, permitting at most $ε-l$ pixels outside $\mathcal{S}$. This abstraction unifies two branching mechanisms: concrete nodes branch through upper shadows to fix additional perturbed locations, while super nodes branch through covering designs to cover many perturbation cases with fewer child nodes. FocusBranch proves node safety by concretizing linear relaxations over node-constrained $\ell_0$-balls and recursively branches on unknown nodes. Rather than relying on a fixed verification plan, it samples nodes across levels to estimate safe rates and costs, then selects a starting level and branching strategy predicted to minimize verification time. Experiments on fully connected and convolutional classifiers for MNIST, Fashion-MNIST, and CIFAR-10 show that FocusBranch verifies $\ell_0$ robustness effectively across architectures and outperforms state-of-the-art verifiers. The code is available at: https://github.com/simpl-group/FocusBranch.

Mon 9 NovMachine Learning
The gist
Checking if neural networks can handle small, sparse changes to input images is tricky because many different pixels could change. The authors introduce FocusBranch, a new way to break down and organize this problem so computers can check safety more efficiently. Their method works by focusing on groups of pixels and smartly exploring possible changes, which speeds up verification. Tests on popular image datasets show it works well and better than previous methods.
Open → 2611.12024v1

Secure fine-tuning of private language model adapters prevents data leaks

VeilTune: Secure Fine-Tuning of Private LoRA Adapters on Public Pretrained Language Models

Abstract: Fine-tuning public pretrained language models with lightweight adapters such as LoRA has become the mainstream way to adapt them to downstream tasks. A service provider often trains its adapter on a data owner's samples, so two assets need protection: the data owner's samples and the provider's adapter. Existing secure fine-tuning protocols protect the samples but reveal to the data owner intermediate results of every training step, such as the hidden representations or logits of its inputs. We show that a single step suffices to copy the adapter: from the hidden representations of one mini-batch of 32 samples, the data owner fits an adapter that agrees with the provider's on 99.2% of test inputs, and logits alone yield about 94%. Protecting both assets therefore requires that no value that depends on the adapter be revealed throughout training. We present VeilTune, to the best of our knowledge the first protocol that fine-tunes a pretrained language model with backpropagation entirely in secure multi-party computation (MPC). VeilTune computes only what the loss depends on, replaces generic MPC operators with training-aware nonlinear operators that stay accurate on the values that training produces, and obliviously skips the steps corrupted by truncation failures. On four few-shot tasks with RoBERTa-large, VeilTune nearly matches plaintext training and is 5.3 points more accurate on average than SecP-Tuning, the state-of-the-art secure fine-tuning protocol. On SST-2, it achieves a 3.5x end-to-end speedup and a 3.6x reduction in communication over SecP-Tuning, and 26x and 17x over full fine-tuning in MPC.

Mon 9 NovCryptography and SecurityArtificial Intelligence
The gist
Fine-tuning public language models often involves using lightweight adapters trained on private data. Existing secure methods keep the data safe but leak intermediate outputs, which can be exploited to copy the adapter. This paper shows that revealing such intermediate results is risky and introduces VeilTune, a new protocol that fine-tunes adapters fully securely without leaking any values related to the adapter. VeilTune performs nearly as well as normal training while being more accurate and faster than prior secure methods.
Open → 2611.11817v1

Large language models gain timing attack defense with Mirage system

Mirage: Provable Defense Against LLM KV Cache Timing Side Channels with Topology-Aware Isolation

Abstract: Large Language Models (LLMs) are increasingly deployed as shared multi-tenant services, relying on advanced optimizations like KV cache and prefix sharing to improve throughput and reduce latency. However, these optimizations introduce a critical vulnerability: attackers can exploit time-to-first-token (TTFT) variations between KV cache hits and misses to infer sensitive prompt tokens via prefix-matching side-channel attacks. Existing defenses indiscriminately restrict cross-tenant cache sharing, which incurs prohibitive performance overhead and negates the fundamental efficiency benefits of the KV cache. We propose Mirage, a topology-aware emission obfuscation framework that provably bounds timing leakage without sacrificing the performance gains of cache sharing. Instead of stalling the GPU or forcing redundant recomputations, Mirage delays token emission at the gateway so the attacker observes a natural cache miss distribution, while the GPU seamlessly continues serving other requests at full speed. Mirage uses a Privacy Odometer that persistently tracks the topological novelty of prefix fan-out expansions to distinguish aggressive probing from benign traffic. Once a prefix exceeds a safety threshold, Mirage engages a two-phase policy that uses a dynamic estimator to pad the TTFT to a calibrated cache-miss baseline. On a 500-sample public corpus across three models, Mirage suppresses the timing side-channel to near the random-guessing floor, while maintaining a significantly lower TTFT than defense baselines. We further show that Mirage withstands a strong adaptive adversary and the same principle carries over to semantic caching layers.

Sat 7 NovCryptography and SecurityArtificial Intelligence
The gist
Large language models (LLMs) use special techniques to work faster, but these can leak secret prompt information through small timing differences. The authors created Mirage, a method that hides these timing clues without slowing down the system. Mirage carefully delays certain responses so attackers can’t tell if data was cached or not, while the model keeps running at full speed. Tests show Mirage reduces this security risk close to random guessing, even against clever attackers.
Open → 2611.10756v1

Memory systems in AI agents vary in vulnerability to poisoning attacks

MemoryDojo: Benchmarking Poisoning Through LLM Agent Memory

Abstract: LLM agents benefit from using memory systems to improve performance on long-horizon, multi-session tasks. However, memory introduces a critical attack surface. Malicious instructions encountered in one session may be stored and propagated through memory before being retrieved and acted upon in later sessions. Because memory systems differ in how they process and retrieve information, the downstream effects of poisoning can vary significantly across systems. To quantify the effects of poisoning across different memory architectures, we introduce MemoryDojo, a benchmark for tracing the lifecycle of poisoned information through agent memory and measuring its influence on subsequent agent behavior. MemoryDojo is easily extensible, consists of 155 multi-session long-horizon trajectories covering four domains, and supports five different memory systems. Through our evaluation, we find that all memory systems are susceptible to memory poisoning, with substantial variation in downstream effects. We further propose a memory-aware poison optimization process that increases the poisoning efficacy, highlighting differences between memory architectures. We hope that MemoryDojo will support research on understanding and mitigating agentic memory risks.

Fri 6 NovCryptography and SecurityArtificial IntelligenceComputation and Language
The gist
Large language model (LLM) agents use memory to handle long tasks across multiple sessions, but this memory can be attacked by harmful instructions that get stored and used later, causing problems. The authors created MemoryDojo, a tool to test how different memory systems handle these attacks by tracking bad information over time and its influence on the agent's decisions. They found that all memory types can be compromised, but some are more resilient than others. They also developed a way to make attacks more effective to better understand the weaknesses in various memory designs.
Open → 2611.09721v1

Acoustic plans help find harmful sounds in text to audio models

AUTEUR: Acoustic Plan Learning for Red-Teaming Text-to-Audio Models

Abstract: Automated red-teaming of text-to-audio models requires test prompts that pass textual screening and expose harmful acoustic content. The central challenge is to connect the sounds produced by the target model to both reusable test generation and prompt revision. Our key idea is to represent test descriptions as structured acoustic plans that organize foreground sounds, context, and vocal attributes. To this end, we propose acoustic plan learning, a framework that learns and refines these plans using waveform feedback. We train the plan generator through supervised fine-tuning followed by reinforcement learning with audio feedback while keeping the target audio model fixed. At inference time, the same trained model generates initial plans and uses feedback to revise their context and vocal attributes while preserving the foreground. Experiments across four target systems show attack success rates of 79.4% on TangoFlux and 95.6% on Kling. The resulting test prompts also transfer across open-source and closed-source models, indicating shared acoustic vulnerabilities.

Fri 6 NovArtificial IntelligenceSound
The gist
It can be hard to check if text-to-audio systems produce harmful sounds because text filters won’t catch everything. The authors create a way to describe sounds as a plan, including main sounds, background, and voice features, then use audio feedback to improve these plans. Their method tests models by making sounds that sneak past text checks but expose harmful audio. This approach found flaws in several systems and worked across different models, showing some common weaknesses.
Open → 2611.09351v1

Existing prompt guards fail to stop attacks on legal contract AI tools

Notwithstanding Anything to the Contrary, Ignore All Instructions: Measuring Prompt Injection Guards on Legal Contract Review Attacks

Abstract: Specialized prompt injection guards are deployed in front of document analysis systems, but no false positive rate has been published for any of them on legal contracts. We measure eleven open-weights guards on two corpora of legal contracts and find that existing guards struggle: document-level false positive rates are as high as 92.39%; two guards fall below chance on paired AUC; and a five-regex baseline is not Pareto-dominated by any guard in the panel. Controlled probes locate model-specific as well as general failure modes. Prompted LLMs resolve the tested failures, including a second threat category (unauthorized scope claims) on which every guard scores zero. A cascaded LLM solution, where the positives of a mid-tier model are verified by a frontier model, is 6.5x cheaper than using exclusively a frontier model and approaches its performance.

Fri 6 NovArtificial Intelligence
The gist
Artificial intelligence tools are sometimes tricked by sneaky instructions hidden in legal contracts. The authors tested eleven guard techniques meant to stop these tricks but found that many failed, with some incorrectly blocking up to 92% of real contracts. They discovered new ways these guards fail and showed that using smart language models to double-check can catch these attacks more reliably and cheaply than just using the best models alone.
Open → 2611.08834v1

Kernel machines show stronger resistance to adversarial attacks than neural networks

Adversarial Robustness of Kernel Machines: A Comprehensive Analysis

Abstract: Kernel machines are an important class of models in machine learning. However, the question of how robust modern kernel machines are to adversarial examples has not been studied comprehensively. In this paper, we provide a thorough empirical investigation of the adversarial robustness of kernel-based models for both binary and multiclass classification tasks, via white-box and black-box evasion attacks. We evaluate the native robustness of kernel machines, random feature classifiers, and the relatively newer Recursive Feature Machines. In addition, we provide an algorithm for approximate adversarial training of kernel machines. Our experiments on classification tasks on image and tabular datasets strongly indicate that kernel machines are natively more robust than neural networks, i.e., their performance degrades more gracefully against adversarial perturbations.

Thu 5 NovMachine LearningArtificial Intelligence
The gist
Some computer programs, called kernel machines, help computers recognize patterns, like images or tabular data. The authors tested how well these programs resist tricky, slightly changed inputs designed to fool them, called adversarial examples. They found that kernel machines are naturally better at handling these tricky inputs compared to another popular kind of program, neural networks. They also developed a way to train kernel machines to become even tougher against attacks.
Open → 2611.07474v1

Detectors struggle to spot rare attacks and unseen tools in hijacked AI agents

What Standard Evaluations Hide About Detectors of Hijacked LLM Agents: Rare Attacks, Unseen Tools and Benchmark Transfer

Abstract: Detectors can flag hijacked large language model (LLM) agents from the traces of the tools they call. In the eleven evaluations we audited, attacks or anomalies make up 16% to 86% of the test samples, no evaluation varies that share, and six report no test on data from a second benchmark or source. We ask what such evaluations hide. On the AgentDrift and AgentDojo benchmarks we test trained detectors that read tool identities, behavior, call sequences, step embeddings or text, and a released LLM action monitor, AlignmentCheck, with thresholds fixed on training data or set by the monitor's released decision rule. First, at a 2% attack rate a detector with an area under the receiver operating characteristic curve (AUROC) of 0.879 reaches a precision of 0.076, and the mix of negatives alone moves that precision between 0.039 and 0.117. Second, as tool coverage, the share of test tool calls whose tool was seen in training, falls, detector rankings cross near 50% coverage and reverse below it, because unseen tools map to inputs that training never reached. Dropout on tool inputs largely removes the reversal. Third, no detector trained on AgentDrift ranks AgentDojo traces better than chance (AUROC 0.47 to 0.52), including one that reaches 0.996 within AgentDojo. The monitor, trained on neither, reaches 0.930, but with its open backbone its released decision rule would still mostly raise false alerts at a 2% attack rate (precision 0.096). We turn these findings into a checklist for evaluating trace detectors.

Fri 30 OctCryptography and SecurityMachine Learning
The gist
Detecting when AI assistants have been hijacked can rely on spotting unusual patterns in the tools they use. The authors found that standard tests give detectors an easier job by including many attacks and familiar tools, which hides real-world challenges. When attacks are rare or tools are new, detectors often fail or make many false alarms. The authors show that detectors trained on one dataset do poorly on another and propose guidelines for better testing.
Open → 2610.41803v1

Gradient guided data augmentation improves model learning from protected images

A Gradient based Adversarial Augmentation Framework for Mitigating Unlearnable Datasets

Abstract: Unlearnable data methods (ULD) protect public data from unauthorized model training by adding imperceptible perturbations that create a learning shortcut. A deep neural network can exploit the shortcut learning path to minimize training loss without learning generalizable features. Intuitively, finding adversarial versions of perturbed images that can substantially increase the model's loss can force the model to learn generalizable features. Existing data augmentation based attacks rely on heuristic search strategies/random sampling over a curated augmentation set to identify adversarial augmentation operations, an approach that often does not scale when the number of candidate operations is large. To address the challenge of a large search space, we propose {\techname}, a gradient based adversarial augmentation framework that learns an augmentation policy via stochastic gradient ascent. {\techname} leverages a differentiable data augmentation pipeline to efficiently search a set of up to 18 augmentation techniques, from which a grid search selects a protection-specific subset. The augmentation subset is trained to maximize a surrogate model's classification loss. We introduce a perceptual-plus-pixel regularization to learn operation parameters that preserve image semantics. Under a controlled ablation, {\techname} consistently outperforms an equivalent random sampling baseline (80.0\% vs. 77.29\% clean test accuracy on EMIN-protected CIFAR-10 for ResNet-18 and 83.28\% vs. 66.45\% on NTGA-protected CIFAR-10 for MobileNet-V2), demonstrating that gradient guidance drives the clean test accuracy recoverability rather than the expanded search space. {\techname} offers an effective and scalable adversarial augmentation framework for stress-testing ULD methods, a critical tool for developing robust ULD methods.

Wed 28 OctMachine Learning
The gist
Some people protect public images by adding tiny changes that trick AI models into learning the wrong things, making the data 'unlearnable.' The authors found a better way to help models learn useful features from such tricked images by using a smart method that finds the best data changes using gradients. This method is faster and more effective than just trying random changes and helps models perform better on images meant to confuse them.
Open → 2610.38173v1

Task format influences fine-tuning success in prompt-safety models

Task Format as a Factor in Adaptation: A Controlled Comparison of LoRA and Full Fine-Tuning for Prompt-Safety Classification

Abstract: Fine-tuning a language model for classification involves many design decisions. Comparisons of low-rank adaptation (LoRA) with full fine-tuning are common but often confounded, and task format is rarely studied. We compare both regimes on prompt-safety classification with a matched 400M encoder-decoder pair (Ettin) across three task formats, with up to 15 seeds per cell and pre-specified analyses. At the pre-specified learning rates, full fine-tuning leads LoRA by 2-3 pp in every architecture and format trained on the label alone. On the encoder, the gap closes once both rates are chosen by the same search. Training LoRA to write a reason after its verdict adds 4.0 pp and closes the gap, but a content-free sentence of equal length does not. Trained models score above a prompted 8B in domain but fall behind it on two of three out-of-distribution corpora, trailing by 8-23 pp on adversarial jailbreaks and flagging 27-66% of benign but harmful-looking prompts as unsafe, against the 8B's 2-3%. Together, the results suggest that the adaptation regime matters less than how each model is trained, its learning rate, and training target, and that in-domain accuracy alone is an incomplete guide to robustness.

Sat 24 OctComputation and Language
The gist
Fine-tuning language models to classify whether prompts are safe depends on various choices, including how the task is formatted. The authors compared two fine-tuning methods, LoRA and full fine-tuning, on the same model and found that full fine-tuning usually performs a bit better, but this can change with different training setups. Having the model explain its decisions improved LoRA's performance. They also showed that good accuracy on familiar data doesn’t guarantee strong performance on harder, out-of-domain examples or adversarial cases.
Open → 2610.33813v1

Batch normalization watermarks vulnerable to data free attacks

On the Vulnerabilities and Data-Free Removal of Watermarks in Batch Normalization Layers

Abstract: Batch Normalization (BN) layers have become a common choice for embedding white-box watermarks in Federated Learning (FL), providing a simple and effective mechanism for model ownership verification and participant identification. However, previous works have focused on robustness against generic model modifications, leaving the security implications of this design largely unexplored. In this paper, we identify fundamental vulnerabilities of BN-based watermarking schemes and propose a practical data-free attack that exploits them. Experimental results on two representative FL watermarking schemes show that the proposed attack consistently removes the watermark with minimal impact on model performance, even without access to representative training data. More broadly, our findings highlight the importance of evaluating DNN watermarking schemes under threat models consistent with Kerckhoffs's principle, rather than relying on security through obscurity.

Tue 20 OctCryptography and Security
The gist
Watermarks in AI models help prove who made them, especially in shared learning projects. The authors found that a common way to hide these watermarks inside certain layers called batch normalization can be easily removed without needing any training data. Their method breaks these watermarks while keeping the model working well. This shows the usual security tricks relying on hiding methods might not be enough.
Open → 2610.28076v1

Quantized large language models show a gap between signed and run code

What You Sign Is Not What You Run: The Verifiability Gap in Quantized LLM Supply Chains

Abstract: Model signing authenticates the full-precision weights W an LLM is released with, yet users run a quantized artifact Q whose producing transformation is not uniquely determined under current disclosure. We separate the non-environment inputs from the execution environment E and measure a quantity that, to our knowledge, has not been measured before: the Independent Verifiability Rate (IVR), the fraction of a popularity-sampled GGUF ecosystem that a third party can re-derive bit-for-bit. Across the 200 most-downloaded Hugging Face GGUF repositories (198 in scope), no artifact or model card records the build environment, so under a recorded-inputs protocol the IVR is 0/198. Yet the artifacts are not irreproducible. On four in-sample repositories that disclose their inputs, the card's information alone reproduces no Q4_K_M file, but two coordinates the card omits - the build's floating-point-contraction setting and, for hybrid models, the per-tensor quantization recipe - re-derive 7/8 target-quant files bit-for-bit on x86 (5/8 on arm64); the fp16 re-derives from the provider's checkpoint, but the calibration imatrix must be shipped and hash-bound, since recomputing it did not match. An older release additionally pins the instruction set and did not re-derive. Reproducing the divergence reported with llama.cpp PR #25353, we show that E is a necessary coordinate and that one compiler setting yields a unique output across the tested platforms at negligible quality cost (+0.06% perplexity). As a security characterization, benign floating-point non-determinism separates from uniformly placed nibble-flip tampering above a detection floor that grows with the environments a verifier must allow, but a targeted embedding edit evades the statistic, so the proposed protocol rests on exact re-derivation; adversarial-fp16 attacks lie outside it. We release the audit data and a re-runnable harness.

Wed 14 OctCryptography and SecurityMachine Learning
The gist
When people use large language models (LLMs), they often rely on signed full-precision versions, but actually run smaller, quantized versions made from them. This paper finds that it is usually impossible to exactly reproduce the quantized models just from the provided information and environment details. The researchers define a measure called Independent Verifiability Rate (IVR) and show it is essentially zero under current practices because the build environment is not recorded. They demonstrate that knowing specific build details can allow exact reproduction, improving trust and security in model distribution.
Open → 2610.19498v1

Imperceptible image changes fool vision language model training alignment

DPOison-VL: Clean Preference Poisoning in Vision-Language Direct Preference Optimization

Abstract: Direct Preference Optimization (DPO) has become a widely adopted approach for aligning vision-language models (VLMs), yet the security of multimodal preference data remains largely unexplored. Existing preference poisoning attacks either corrupt explicit supervision signals or require visible triggers, making them detectable under human inspection. We introduce DPOison-VL, the first clean preference poisoning attack against VLM-DPO, which manipulates model alignment solely through imperceptible perturbations on a small fraction of training images, while leaving all prompts, responses, and preference annotations semantically intact. We formulate poison construction as a bilevel optimization and derive a gradient alignment surrogate that efficiently steers image perturbations toward the attacker-specified preference. We further introduce a preference displacement score for principled poison selection with provable first-order optimality under a fixed poison budget, and derive a sufficient condition for preference inversion. Extensive experiments across diverse attack tasks, different VLM architectures and scales, and baseline comparisons demonstrate that DPOison-VL achieves up to 98\% ASR with only 2\% poisoned data and exhibits cross-model transferability, while largely preserving clean accuracy and general utility. Our findings reveal a critical vulnerability in VLM-DPO, where imperceptible image perturbations in preference training data can silently redirect alignment.

Tue 13 OctCryptography and SecurityArtificial IntelligenceComputer Vision and Pattern Recognition
The gist
Vision-language models learn to understand images and text together by training on preferences about which responses are better. This paper shows that subtly altering a small number of training images with invisible tweaks can trick these models into aligning with attacker-chosen preferences. The attackers do not need to change any text or obvious signals, making this kind of attack hard to notice. The authors developed a method to design these small changes efficiently and tested it to show it works across multiple model types with little effect on normal accuracy.
Open → 2610.18609v1

Vision language models reveal new clues to spot adversarial images

Detecting Adversarial Images through Response Profiles of Vision-Language Models

Abstract: Adversarial perturbations can alter the predictions of frozen vision-language models (VLMs) while leaving their confidence and image--text similarity patterns seemingly plausible. We investigate whether we can identify adversarial inputs based on the broader way an image interacts with a collection of general semantic prompts. Our detector summarizes these responses using category-level statistics, relationships among prompts, deviations from clean reference distributions, and stability under weak image transformations, producing a compact response profile that is classified by a lightweight model while the VLM remains fixed. We evaluate the approach on multiple public image datasets, several CLIP-style visual backbones, and a range of gradient-based, optimization-based, automated, and spatial attacks. The detector achieves strong discrimination in attack-specific settings and retains substantial performance when evaluated on attacks not seen during training. Under a controlled detector-specific protocol, the response-profile representation outperforms the evaluated embedding-geometry baselines. Additional analyses show that the feature groups provide complementary information and that the method remains effective under variations in the prompt configuration. We also examine inference cost and performance against detector-aware adaptive attacks. Overall, the results indicate that response patterns across semantic prompts provide a useful complementary signal for adversarial image detection in frozen VLMs.

Wed 7 OctComputer Vision and Pattern Recognition
The gist
Adversarial images are specially altered pictures designed to trick AI models into making wrong predictions without looking suspicious. The authors found that by looking at how a frozen vision-language model responds to many different text prompts, they can detect these tricky images better. Their method creates a summary of these responses and uses a simple classifier to decide if an image is adversarial. This technique works well on various attacks and even when facing attacks it hasn't seen before.
Open → 2610.10436v1

Backdoor attacks in fine-tuned large language models reduced effectively

Backdoor Purification for LoRA-Tuned LLMs via Null-Space Projection

Abstract: With the rapid adoption of large language models (LLMs) and parameter-efficient fine-tuning (PEFT) methods, the risk of backdoor attacks has become more severe. Existing backdoor purification methods typically rely on at least one of the strong assumptions, such as prior knowledge of triggers, access to clean references, or aggressive retraining, and they often lack comprehensive evaluations. These constraints substantially limit their practical applicability. To overcome these challenges, our work proposes purifying LoRA-tuned LLMs without these assumptions and even without post-hoc retraining of the suspect parameters. Our objective is to significantly reduce the attack success rates (ASR) while preserving both (i) the base model's general capabilities and (ii) the new downstream skills learned through the adapter. Through a series of ablation studies, we progressively scale our approach from a single layer in a text classification setting to a full-parameter LLM in the generative task. Through careful data curation and feature approximation, we extract high-fidelity backdoor directions and, for each layer or head, construct orthogonal null spaces in both the input and output channels, onto which the LoRA updates are projected. Empirically, our null-space projection method reduces the ASR from nearly 100% to less than 10%, while preserving the base model's benign performance and the adapter's learned abilities during downstream task adaptation.

Wed 30 SeptArtificial IntelligenceCryptography and SecurityMachine Learning
The gist
Large language models can be tricked by hidden backdoors that cause them to act badly when certain secret inputs appear. The authors introduce a way to clean these models without needing to know the secret backdoor triggers or using extra training. They do this by finding directions related to the backdoor behavior and then removing those influences while keeping the model's normal skills intact. Their method greatly reduces the chance of the backdoor working, while still allowing the model to do its usual tasks well.
Open → 2610.00685v1

Decentralized method detects and blocks backdoors in shared language models

Backdoor Mitigation in Decentralized LLM Fine-Tuning

Abstract: Decentralized large language model (LLM) fine-tuning lets organizations collaboratively train a shared LLM on data they cannot pool, without a central coordinator. In every round, each node exchanges a trainable adapter with its neighbors over a communication graph, and then aggregates them. This setting, however, is vulnerable to propagated backdoors, which is a hidden behavior that lets a model perform normally on clean inputs but produce an attacker-chosen output whenever a secret trigger appears. We show that a single node poisoning its own model can backdoor adapters of nodes that have never seen a poisoned example, making them refuse prompts that contain a secret trigger. We present Chorus, a decentralized mechanism that lets each node detect and reject backdoored adapters from its neighbors before aggregation, without requiring shared validation data or any knowledge of the attacker's trigger or target. Chorus judges each adapter by its behavior, using the receiver's own adapter as a trusted reference. Crucially, no node in Chorus judges adapters alone: the receivers of each adapter update probe it independently, pool their findings in the neighborhood, and vote to make a decision. So a backdoor that slips past one receiver is still caught by the others. We evaluate the effectiveness of Chorus using two instruction-tuning datasets and LLM architectures, and against a state-of-the-art baseline. Chorus cuts the average attack success rate (ASR) of the attacker's neighbors from 48-63% to at most 2.2%, within 0.6 percentage points of an omniscient oracle that knows the exact malicious nodes. Even the worst-affected honest node never exceeds 10% ASR, the same bound as the oracle, against up to 78% without defense. This all comes at a negligible communication overhead.

Tue 29 SeptCryptography and SecurityMachine Learning
The gist
When organizations train shared language models together, some can sneak in hidden backdoors that make the model behave badly when a secret trigger appears. The paper shows that even one bad participant can affect many others who never saw the bad data. The authors propose Chorus, a way for each participant to spot and reject these backdoors by working together and comparing the behavior of updates before accepting them. Their tests show Chorus drastically reduces attacks without extra data or revealing secrets.
Open → 2609.37367v1

TraceGuard detects poisoned data in image text training datasets

TraceGuard: Adaptive Multimodal Poison Filtering through Cross-Feature Rank Agreement

Abstract: Multimodal training relies on image-text corpora collected from external sources, creating opportunities for attackers to poison the data. Stealthy attacks can preserve plausible image-text pairs while concealing the differences used by detectors, so apparently clean data can still redirect the trained model. We therefore ask which properties a poison set must preserve for the attack to remain effective. A small poison set must still exert enough collective influence during training to induce the attacker's target behavior. We analyze this influence in terms of how often an attack pattern occurs and how strongly the examples carrying it jointly affect the model. This analysis motivates six corpus-level features that examine cross-modal neighborhoods, recurring text, and changes after text-span erasure without training the victim model. We introduce TraceGuard, an adaptive rank-based filtering method that uses agreement among complementary feature rankings to identify suspicious examples. It refines the selected set through shared patterns and adapts the removal threshold to each corpus without knowing the attack or poison rate. Across 19 attack configurations spanning image-text learning, generative vision-language model fine-tuning, and encoder-transfer tests, TraceGuard removes an average of 98.4% of poisoned examples and 5.4% of clean examples. After training on the filtered corpora, the residual attack metric is at most 1% in 13 configurations. Matched-removal controls and ablations support the contributions of sample selection and adaptive removal. Stress tests also identify detection failures under adaptive attacks and unnecessary removal on poison-free corpora.

Thu 24 SeptCryptography and SecurityMachine Learning
The gist
Training AI models with images and text from the internet can be risky because attackers can hide harmful examples that trick the model. The authors study what makes a small set of bad data powerful enough to affect training. They develop TraceGuard, a tool that spots suspicious examples by looking for agreement among different patterns in the data without needing to train the model first. Their tests show TraceGuard removes most bad examples while keeping most good ones, making AI models safer to train.
Open → 2609.29099v1

Language models balance fast output and traceable text with new sampling method

Watermarkable Multi-Draft Speculative Sampling via Poisson Processes

Abstract: Large language models (LLMs) have achieved state-of-the-art performance across a wide range of tasks, motivating two important aspects of deployment: inference efficiency and output provenance, which can be tackled by speculative sampling and watermarking, respectively. However, recent works have shown that combining these two goals is highly nontrivial and can be potentially impossible. In this work, we develop a novel multi-draft speculative sampling algorithm based on Poisson processes that improves the frontier of this fundamental trade-off. The proposed algorithm has strong sampling efficiency on its own and, more interestingly, is naturally watermarkable: we can embed an unbiased watermark without degrading speculative acceptance. Moreover, our algorithm is based on an exact list-coupling-without-communication scheme, which yields a drafter invariance property that benefits both sampling and watermarking. It is the first multi-draft, drafter-invariant speculative sampling scheme that maintains both watermark strength and sampling efficiency, and we experimentally verify its strong performance in both aspects.

Fri 18 SeptCryptography and SecurityMachine Learning
The gist
Large language models can create text quickly and let people know that the text came from them. But doing both at the same time is hard. The authors designed a new way to pick words using a process based on random timing, which helps keep the text fast and easy to check. Their method also makes sure the text carries a special signature (a watermark) without slowing things down. They tested their idea and found it works well for both speed and watermark quality.
Open → 2609.21858v1