BLOOM-WILT: Logit Tilting for Behaviour Elicitation in Automated LLM Auditing
2026-08-31 • Artificial Intelligence
Artificial IntelligenceComputation and Language
AI summaryⓘ
The authors explain that language models often show rare behaviors only after many uses, which regular testing misses. They created BLOOM-WILT, a new way to audit these models that finds unusual behaviors more efficiently without needing extra training or internal access. Their method learns from previous interactions and changes how it samples the model’s responses to find these behaviors faster. When tested on several models and behaviors, BLOOM-WILT performed better than older methods, especially at detecting harmful outputs. This tool helps better understand and assess language models in realistic use.
language modelautomated auditingelicitationmulti-turn conversationsamplingnext-token distributionself-harm encouragementmodel safetyscore-based learningdecoding reweighting
Authors
Adrians Skapars, Edoardo Manino
Abstract
Users of a deployed language model routinely encounter behaviours that testing almost never surfaces, since deployment puts the model through orders of magnitude more interactions than any evaluation can simulate. Automated auditors make testing cheap to scale and flexible enough to cover almost any specified behaviour, yet their lack of optimisation pressure makes them sample-inefficient. To address this shortcoming, we introduce BLOOM-WILT, a full auditing pipeline that elicits natural multi-turn instances of rare behaviours, without training cost or access beyond the target's next-token distribution. On the input side, WILT's auditor model revises its conversational strategy across rounds, learning from previous scored interactions. On the output side, WILT adaptively reweights the target's decoding using the model's own distribution conditioned on an elicitation prompt, so that behaviour-relevant generations are sampled ahead of others it finds equally probable when unprompted. We evaluate WILT across 4 target models and 8 behaviours, where it beats the baseline auditor in 30 of the 32 settings and overturns the previous model safety rankings. WILT raises average behaviour presence from 51% to 100% when eliciting self-harm encouragement from Qwen3.5-4B, beating every elicitation method we port into the same pipeline at matched compute, without pushing output probability below the baseline's.