Transcoders for Investigating Deception in Language Models
Artificial Intelligence
Summary
The authors studied a method called transcoders to understand how language models might produce deceptive responses. They used a specific model and transcoders to map out how features inside the model connect and influence each other. By doing this, they found features linked to deception and showed these features strongly impact when the model gives deceptive answers. Their work suggests deception happens because of how the model's internal parts work together and shows transcoders can help detect risky behavior early.
Authors
Darius Lim, Nathan Leow, Xin Wei Chia
Abstract
Transcoders have recently emerged as a promising approach for mechanistic interpretability (MI), enabling circuit-level analysis of model behaviour. In this paper, we investigate the use of transcoders to analyse deceptive behaviour in language models, a behaviour that poses a safety and security risk. Using a Qwen3-4B model with pre-trained transcoders, specifically per-layer transcoders (PLTs), we construct attribution graphs that capture feature activations and inter-feature dependencies, allowing circuit-level analysis of deception. Through feature steering and circuit analysis, we identified a dictionary of deception-related features and show that these features exert a stronger influence on deceptive outputs, as they produce predictable shifts between deceptive and non-deceptive responses. These findings suggest that deception emerges from internal model mechanisms and highlight the potential of transcoders for behavioural monitoring and early detection of security vulnerabilities related to malicious behaviours in language models.