Autoencoders pinpoint key tokens for efficient language threat explanations
Selecting The Most Informative Tokens in Natural Language Autoencoders
Computation and LanguageArtificial Intelligence
Summary
Explaining why a language model behaves a certain way can be very slow if every word it uses is examined. The authors studied which words or tokens in a model's explanation are most important to spot potential security threats like hidden commands. They found that looking at just a small fraction of the tokens—around 5%—can give nearly the same understanding as looking at all of them. They also showed that pre-trained tools can still find words models try to hide, even without extra training for those specific hidden words.
What this means in practice
- •For security audit teams: Generate focused explanations by inspecting a small subset of tokens to efficiently detect language model manipulation or hidden threats.
- •For chatbot developers: Improve debugging by targeting specific token positions to explain automated model decisions and ensure safer conversational agents.
Authors
Federico Torrielli, Gianluca Barmina, Andrea Blasi Núñez, Amon Rapp, Luigi Di Caro, Peter Schneider-Kamp, Lukas Galke Poech
Abstract
Natural language autoencoders translate a language model's internal activations into readable explanations. Explaining every token position is costly. Which positions should an auditor inspect to understand a potential threat? We study this question across $4.7$ million explanations on prompt injection and concealment. We compare signals from model computation with a ranker trained only on chat structure. Chat structure usually selects more relevant explanations than the computational signals, without requiring a model forward pass for position selection. On three of four datasets, explaining just $5\%$ of positions retains nearly all of the success rate from explaining every position, where success means obtaining an explanation about the threat. The benefit varies with the audit task. We also show that pretrained verbalizers recover words that models have learned to conceal through fine-tuning, without additional verbalizer training. These results identify where auditors can concentrate explanation generation and show that useful explanations can extend beyond the model a verbalizer was trained to describe.