The First Token Is a Clue: Verbalizing Multi-Token Concepts from the J-lens
2026-08-31 • Computation and Language
Computation and Language
AI summaryⓘ
The authors studied how to better understand multi-word ideas inside language models using the Jacobian Lens (J-lens). They found that while J-lens can identify the first word of a phrase well, the language model itself can often predict the following words when given this first word. By combining these insights, the authors extracted vectors (mathematical representations) for whole phrases, improving how well they can read and manipulate multi-word concepts compared to earlier methods. Their approach worked better at recognizing and swapping these concepts within models without changing the model’s parameters.
Jacobian Lenslanguage modelshidden statesmulti-token conceptsvocabulary tokensvector representationfrozen modelphrase reconstructionmodel interpretabilitycausal intervention
Authors
Xijie Gong, Tonghan Wang
Abstract
The Jacobian Lens (J-lens) is a recent tool for interpreting LLMs. It reads a hidden state as a ranked list of vocabulary tokens, leaving multi-token concepts without a representation of their own. The original J-lens work addresses this limitation with Template Lens, which precomputes vectors for a fixed phrase vocabulary, and Oracle Lens, which fine-tunes components to propose phrases and reconstruct phrase vectors. We ask whether multi-token concepts and their vectors can instead be recovered directly from J-lens and the frozen model. We find that the first token of a multi-token concept is about as readable as a single-token concept. Given the correct first token and source prompt, the frozen model recovers the second token in 88.3% of two-token cases. We show that a vector for the complete concept can be recovered from subsequent hidden states in a single forward pass. We therefore use J-lens to propose first tokens and let the frozen model complete candidate concepts. We then recover a vector for each candidate and score it alongside the complete vocabulary. Across 496 multi-hop clozes on Gemma-3-12B-IT, Llama-3.1-8B, and Qwen3-14B, our method achieves an average $\mathrm{Rank@}10$ of 43.1%, compared with 27.6% for Template Lens. Without the J-lens clue, performance drops to 21.6%, showing that the first-token clue substantially improves readout. Causal concept swaps using the recovered vectors achieve an average $\mathrm{succ}@10$ of 61.4%, compared with 26.2% for Template Lens under the same intervention. These results show that first-token clues can guide multi-token concept recovery, while subsequent hidden states provide vectors for readout and intervention.