Predicting how well relations can be decoded without knowing the relation
Predicting Linear Relational Decodability Without the Relation
Computation and Language
Summary
Some language models try to find simple math rules to connect one fact to another, but this works better for some types of facts than others. The authors found they can guess how well these rules will work without looking at the specific fact types by examining what the model already knows about the people or places involved. They introduced a method called the 'entity lens' that focuses on the kind of entity instead of the relation. Their findings show that understanding how facts are stored in the model helps explain why some connections are easier to find than others.
What this means in practice
- •For language model developers: Evaluate which relations in a language model are likely decodable with simple linear methods without needing relation-specific info.
- •For data annotation teams: Identify and address problems like semantically weak tokens and ambiguous entity names in relation datasets to improve data quality.
Authors
Jonas Rohweder, Adrian Mülthaler, Farid Mehri, Sebastian Gerstner, Hinrich Schütze
Abstract
Linear relational embeddings (LREs) approximate a language model's subject-to-object mapping with an affine function. Their faithfulness varies enormously across relations, from near zero for person_father to the ceiling for country_capital_city. We investigate the factors that contribute to this variation and demonstrate that it can be significantly predicted from a readout that is not aware of the underlying relation. To do so, we propose a novel form of the Jacobian lens, the entity lens. Whereas the Jacobian lens is estimated across all tokens and concepts, the entity lens EL_t is trained to predict what the model will output for an entity of type t under the neutral prompt "{} is known for"; e.g., we train ELs for countries and persons. After estimating an entity lens, we measure how highly it ranks the model's answer for a relation without using any relation-specific information in estimation and prediction. Across 28 relations of the LRE dataset on Olmo-3-7B this relation-agnostic score correlates with LRE faithfulness at ρ= 0.7355 (p < .01). This suggests that linear decodability is largely a property of what a subject's representation already carries, rather than of the relation being decoded. Moving the unit of analysis from the relation to the individual fact, we conclude that new relation datasets and analysis methods are needed to address several problems in existing datasets, most prominently semantically weak first tokens (e.g., most predicted universities' first token is "University") and reference competition (e.g., the LLM confuses obscure actor names with competing non-actor entities that have the same last name). Our full pipeline is available at https://github.com/jonasrohw/entity-lens