Late layers in large language models copy entities by attending to context
Late Attention Layers Alone Can Copy Entity Tokens, but Not Without Attending to Their Context
Computation and Language
Summary
Large language models can repeat names and other important bits from a question when they answer it. This paper looks at which parts of the model do this copying. The authors find that the later parts of the model are needed to copy exactly the right words and that paying attention to the words around the name helps the model copy better. This helps us understand how language models use context to remember and repeat information.
What this means in practice
- •For language model engineers: Identify specific layers to optimize for entity copying to improve question answering accuracy.
- •For chatbot developers: Design chatbots that better maintain named entities in conversation by controlling attention to context tokens.
Authors
Muyu He, Yuchen Liu, Ran Tao, Li Zhang
Abstract
Large language models (LLMs) reliably perform entity copying, in which a model copies tokens referring to an entity, termed entity tokens, from the prompt into its output to answer a question. Although entity copying is straightforward for most LLMs, existing research does not provide a systematic account of which layers specialize in this fundamental task or how other tokens in the same sequence, termed context tokens, influence the model's ability to copy the entity tokens. To address these questions, we conduct experiments on Qwen3-8B using two novel methods: genie-in-a-bottle, which controls exactly which layers can participate in an entity-copying task, and attention lobotomy, which cuts off specific tokens' attention to entity tokens without affecting the remaining attention distribution. We find that two distinct groups of layers in the second half of the model are both necessary and sufficient for entity copying. Moreover, in addition to the decoding position's attention to entity tokens, context tokens' attention to entity tokens also proves necessary for copying the exact tokens, even though context tokens do not store entity information themselves unless they satisfy particular semantic properties. Our findings establish the critical role of late layers in entity copying under the guidance of context tokens, calling for future work on how models propagate and consume entity information.