Papers for

nlp engineers

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

Models struggle to know nigeria facts even in english

NaijaFacts: A Framework for Disentangling Language from Knowledge in Facts about Nigeria

Abstract: Multilingual benchmarks for African languages mostly translate generic questions into local languages, measuring linguistic competence but not whether models know facts about Africa. We introduce \naija, a benchmark of factual and statistical questions about Nigeria, authored by native speakers and grounded in authoritative official statistics, with every question available in English, Nigerian Pidgin, \yoruba, Igbo, and Hausa. Answering similar questions for the UK and US yields a country by language matrix that disentangles a language effect (the model knows the fact but degrades in a low-resource language) from a knowledge effect (the fact is absent in every language). Across eight frontier and Nigeria-adapted models, the knowledge effect dominates: English-query accuracy is 49--50\% for the US/UK but 20\% for Nigeria. The language effect is smaller and tracks web representation, degrading from English to \yoruba{} (9 points on Nigeria, up to 24 on US/UK questions). We find scale does not help: the strongest models reach 66--79\% on the UK and the US but none exceed 31\% on Nigerian statistics. Extending to all 54 African countries with our best performing frontier model shows the same profile continent-wide, 96\% on culture and 98\% on politics against 26\% on statistics, so the gap is not specific to Nigeria.

Wed 4 NovComputation and LanguageArtificial IntelligenceMachine Learning
The gist
Most language tests measure how well models understand the language, not whether they know true facts about Africa. The authors created NaijaFacts, a test with true facts about Nigeria in several local languages and English. They found that models often do not know Nigerian facts, even when asked in English. This lack of knowledge is much bigger than any drop caused by language differences. The problem is common for many African countries, not just Nigeria.
Open → 2611.06504v1

Causal optimal-transport attention enables longer context length extrapolation

Causal Optimal-Transport Attention: Positional Priors and Length Extrapolation

Abstract: Causal masking obstructs the column balancing used by optimal-transport (OT) attention. We study a prefix-consistent unbalanced operator and characterize the positional prior it induces. Under bounded scores, for any fixed balancing strength $ρ<1$, its fixed point dilutes attention: every fixed window receives a vanishing share as the context grows. For constant scores, we derive an exact scale-invariant prior. We then classify positional encodings by whether their distant attention mass remains bounded away from one. Discrete window-mass bounds quantify the finite-length dilution of summable log-distance priors near exponent one. Preregistered experiments with 25M- and 100M-parameter language models establish improved loss extrapolation over softmax; exploratory comparisons favor an explicit distance bias at a fraction of the cost. CABLE outperforms RoPE with evaluation-time YaRN on every seed at three training lengths. Changing only the bias beyond the training length moves long-context loss in the predicted direction on all 30 tested checkpoints. Even with exponents constrained above one, many heads approach the summability boundary during training. A margin selected by prior-only calculations for a specified window, length range and growth tolerance controls distant-mass growth in the tested setting. These results connect loss extrapolation to positional decay over the lengths actually evaluated.

Sun 1 NovMachine Learning
The gist
Long-range attention in AI language models often struggles when trying to understand longer texts than those seen during training. This work studies a specific type of attention called causal optimal-transport attention, which respects the natural order of words but has challenges balancing attention across long sequences. The authors analyze how this method spreads focus over positions and show ways to control the attention so that models better handle longer texts without losing important context. They test their approach on language models and find it improves predictions on longer inputs compared to standard methods.
Open → 2611.01472v1

Online prediction reduces verification calls for nested text evidence

Online Evidence-Containment Prediction over Text Spans under Distinct Queries

Abstract: Evidence-grounded NLP systems often verify nested text regions sequentially, so verifying one region constrains others. We formalize this as Online Evidence-Containment Prediction (OECP): each candidate is predicted before exact verification and queried at most once. Span containment is exactly a triangular 2D product order, making OECP distinct-query online isotonic regression. With multiple admissible targets, worst-case regret is near-linear, $\widetildeΘ(m_{\mathrm{eff}})$, in the effective scale given by endpoint-rank compression; a single fixed target, as in standard QA, admits an explicit polynomial-time learner with optimal $Θ(\log m_{\mathrm{eff}})$ regret, run on real NQ episodes. On Natural Questions, containment yields a large deductive gain plus a smaller learned gain on unresolved candidates, reproduced across two backbones and DPR hierarchies. Expected Search-space Reduction (ESR) turns these probabilities into decisions, localizing evidence in fewer verifier calls on SciFact, ERASER, and DPR hierarchies; on DPR hierarchies under call budgets it outperforms four recent external selectors and raises fixed-reader F1 over learned and generator-greedy scheduling.

Fri 30 OctArtificial Intelligence
The gist
Finding relevant evidence in large texts often requires checking many possible parts, one after another, which takes time. This paper studies how to predict if one piece of text contains another before fully verifying it, based on special ordering rules. The authors created a method that makes these predictions efficiently and tested it on real-world question-answering data, showing it can cut down the amount of checking needed. This helps systems find the right evidence faster and more accurately.
Open → 2610.41384v1

Hierarchy aware method improves table data understanding in scientific papers

HARP-LoRA: Hierarchy-Conditioned Rank Projection for Low-Rank Adaptation in Scientific Table Understanding

Abstract: Tabular data is widely used in scientific applications, where complex layouts, including multi-level headers, merged cells, captions, and footnotes, make information extraction difficult. This is particularly important in pharmacokinetics (PK), where values must be correctly associated with the drug, species, dose, administration route, unit, and experimental condition. Parameter-efficient fine-tuning methods such as Low-Rank Adaptation (LoRA) reduce adaptation cost but do not explicitly incorporate relationships between table values and surrounding headers. We introduce HARP-LoRA, a hierarchy-conditioned rank projection method for low-rank adaptation in scientific table understanding. HARP-LoRA extends projected LoRA by constructing token-specific intrinsic representations from the row- and column-header hierarchies of each data token. These representations condition the low-rank down-projection, incorporating hierarchical context while keeping the pretrained model and random projection tensors frozen. We evaluate HARP-LoRA on 950 manually annotated PK tables from 587 scientific articles and on WikiTableQuestions (WikiTQ) using Llama~3.2 3B, Llama~3.1 8B, and Gemma~3 27B. With Llama~3.1 8B, HARP-LoRA achieves a PK F1-score of 57.64\% and WikiTQ accuracy of 53.15\%, improving over LoRA by 1.42 and 6.56 points, respectively. With Gemma~3 27B, it achieves the best PK F1-score of 70.63\% and WikiTQ accuracy of 56.87\%. These results use only 172K, 197K, and 547K trainable parameters, far fewer than LoRA and TableLoRA. Overall, conditioning lightweight adapters on table-header hierarchies improves extraction from scientific tables and reasoning over flatter general-domain tables while maintaining parameter efficiency. Code is available at: https://github.com/hosseinsholehrasa/HARP-LoRA

Sun 25 OctComputation and LanguageArtificial IntelligenceInformation Retrieval
The gist
Scientific tables often have complicated layouts with headers and merged cells that make it hard for computers to understand the information inside. This is especially true in fields like pharmacokinetics, where it's important to match numbers to the right drug, species, or dose. The paper presents HARP-LoRA, a method that helps computers learn these relationships by using the table’s header hierarchy without changing the main model. The authors show that this approach improves accuracy on both scientific and general tables, while keeping the model efficient and using fewer adjustable parameters.
Open → 2610.34589v1

Synthetic linguistic data reveal insights into language use and models

Synthetic linguistic data

Abstract: Synthetic linguistic data have become increasingly prominent in linguistic research, from the analysis of large collections of AI-generated texts to controlled experiments investigating grammatical and lexical preferences. This glossary entry extends Bresnan's (2016) distinction between the linguistic garden and the bush by introducing an artificial plant nursery as a metaphor for synthetic data. We distinguish between synthetic bush data, consisting primarily of corpus-like collections of model-generated language, and synthetic garden data, in which model outputs are elicited under controlled conditions to investigate specific linguistic hypotheses. We argue that synthetic data constitute a distinct object of linguistic investigation: comparing them with natural language data can reveal both shared patterns and divergences, but their interpretation requires careful consideration of the conditions under which they are generated.

Sun 25 OctComputation and Language
The gist
Linguistic research often uses language data, but synthetic data generated by AI models are now becoming common. The authors explain how this synthetic data can be like an artificial plant nursery, divided into two kinds: collections that mimic natural language and controlled experiments that test specific language ideas. They show that studying synthetic data alongside real language data helps find similarities and differences, but understanding their meaning requires knowing how the data were produced. This approach adds a new way to study language using AI-generated examples.
Open → 2610.34215v1

Combining multiple smaller language models improves large model reasoning accuracy

Multiple Importance Sampling for Guided Speculative Inference

Abstract: We study an extension of the Guided Speculative Inference (GSI) algorithm for test-time-scaling of large language models (LLMs) to a novel setting where an ensemble of smaller LLMs proposes draft reasoning steps which are later verified by a larger base model and a process reward model. Using the established framework of multiple importance sampling (MIS), we introduce weighting schemes to aggregate samples from a diverse collection of proposal distributions. We provide new bounds on the KL divergence from the optimal reward-tilted target distribution and the expected reward degradation, then demonstrate that the well-known balance heuristic is optimal. Compared to GSI with a single well-tuned drafter, combining several diverse LLMs while retaining the same total number of samples maintains competitive pass-at-$N$ downstream performance while being mathematically robust to situations where the support of a single draft model does not cover that of the target, where GSI may fail.

Fri 23 OctMachine LearningInformation Theory
The gist
The paper looks at how to improve the way large language models (LLMs) solve problems by using several smaller LLMs to suggest possible reasoning steps. These smaller models each come up with ideas, and then a bigger model checks and scores these ideas to find the best answers. The authors use a mathematical method called multiple importance sampling to combine the suggestions in a way that balances their contributions. They show this approach keeps performance strong while avoiding problems when one model’s suggestions don’t fully cover the needed reasoning steps.
Open → 2610.33368v1

Dreaming learning helps keep generative models diverse and stable

Mitigating Model Collapse through Dreaming Learning

Abstract: Training generative models on synthetic data causes loss of diversity and a decrease in performance, a phenomenon known as model collapse. We study this degeneration spectrally in a controlled Markov setting. The eigenvalue spectrum of the learned transition operator offers a direct view of the process, since the subdominant eigenvalues set how fast the chain mixes and therefore how much of the state space it continues to explore. We find that sampling at temperature $T_s < 1$ acts on the operator like repeated sharpening, pushing it toward a deterministic system: the subdominant eigenvalues approach the unit circle and the spectral gap narrows. When $T_s < 1$, even data accumulation does not appear to arrest this drift. We then analyze Dreaming Learning, a regulariser that, in parallel with regular training, adds a loss term on the model's own samples drawn at a raised temperature $T_d > 1$. Because high-temperature samples flatten the predicted distribution, training on them keeps the entries of the learned operator from becoming vanishingly small, which in turn keeps the eigenvalues bounded away from the unit circle. Measurements of the spectra over 5000 generations are consistent with this picture, and the mitigation effect persists in more complex settings such as LSTM language models and a 124M-parameter GPT-2.

Wed 21 OctMachine Learning
The gist
Generative models can lose diversity and perform worse when trained on their own synthetic data, a problem called model collapse. The authors studied this problem by looking at the math behind how the models explore possible outputs. They found that sampling with lower randomness makes models more deterministic and more prone to collapse, but training on their own outputs created with higher randomness helps the model stay diverse. This approach, called Dreaming Learning, keeps the model's internal math balanced and prevents collapse, even for large language models like GPT-2.
Open → 2610.30009v1

Llms judge generic statements similarly to humans but rely on factual truth

Why Do LLMs Accept Generics? Explaining Acceptability using Psycholinguistic and Model-Derived Factors

Abstract: Psycholinguistic studies explore why humans accept some generic statements but reject others by showing that these judgments depend on more than how common a property is. People consider several other factors such as how well it sets the kind apart from others or whether it is characteristic of the kind. Inspired by the linguistic and cognitive literature, we explore to what extent factors from these studies explain how large language models (LLMs) judge generics. Our results with twelve models on a psycholinguistic dataset and on naturally occurring generics show that models weigh how common a property is against how well it distinguishes the kind, and combine the two much as people do, though less clearly. Like people, they favor characteristic properties, but unlike the accounts of human judgment, they ignore striking ones. What matters most to LLMs is whether a statement is factually true, and much of the influence of human factors reaches their ratings through that judgment. In six 7-9B models, shifting the hidden states along each factor's direction moves the judgment as the factor predicts.

Tue 20 OctComputation and Language
The gist
Some statements about groups, like "birds fly," are natural for people to accept, while others feel odd. The authors studied why big language models also accept some generic statements but not others, using ideas from how humans think about these sentences. They found that the models consider how common a trait is and how well it defines the group, much like people do, but focus more on whether the statement is factually true. Changing parts of a model's internal data in directions tied to these factors changes its judgment, showing these ideas influence the model's decisions.
Open → 2610.27928v1

Consistency training reduces cue use without teaching when cues are correct

Consistency Training Suppresses Cue Following Without Teaching Cue Selectivity

Abstract: Language models often change answers because of a cue, a hint such as "I believe the answer is B" that gives no valid evidence. Consistency training (CT) trains a model to answer the same with and without the cue. Four CT methods apply this idea to output tokens (BCT), residual activations (ACT), attention patterns (AttCT) and sampled-answer rates (RMCT). Prior work suggests some learn different solutions. We train all four on Gemma-2-2B-it and Llama-3.1-8B-Instruct with the same data. We test predictions written in advance and inspect the models with a known circuit that carries cues to the answer. All four make the trained cue less effective, and their fixes share three properties. First, correct cues lose about as much influence as wrong ones. Second, the fix is usually large for cues before the answer options, as in training, and much smaller after them. Llama RMCT depends least on position. Third, the circuit heads still carry most remaining cue-induced errors. Making BCT targets depend on the cue's correctness mostly weakens the fix. CT thus weakens the trained cue without teaching models when a cue is right. Evaluations with only wrong cues at the trained position miss the first two properties.

Mon 19 OctComputation and LanguageArtificial IntelligenceMachine Learning
The gist
Language models sometimes change their answers based on extra hints that don’t actually provide real information. The authors studied methods that teach models to give consistent answers with or without those hints. They found that while these methods do lessen the effect of these hints, they don't help models learn to tell when a hint is actually helpful or not. Instead, the hint’s influence weakens equally whether the hint is right or wrong. This shows that current training methods suppress cue-following but don't improve the model’s understanding of cue reliability.
Open → 2610.25650v1

Large language models improve knowledge use with efficient chunk memory

Deep Fusion Memory: A Chunk-Level Parametric Memory for Large Language Models

Abstract: Large language models need effective ways to acquire and use external knowledge beyond what they retain from pretraining. Continued pretraining risks catastrophic forgetting, while retrieval at inference time can add substantial overhead. External parametric memory avoids online retrieval, but constructing its training data requires extensive offline search, and output-level interpolation provides only shallow fusion. We introduce Deep Fusion Memory (DFM), a parametric memory architecture that reduces this construction workload and incorporates learned memory into intermediate hidden states. A causal memory predictor learns to generate memory from the preceding context, using retrieved chunk embeddings from the corpus as supervision. The predicted memory is shared across tokens within each chunk and fused into their hidden states through gated cross-attention. Memory prediction and fusion are trained end to end with the backbone frozen, and inference requires no explicit retrieval. Compared with existing parametric memory approaches, which often construct token-level memory targets, DFM requires $32\times$ fewer retrieval queries and $2048\times$ fewer retrieved items to construct its training data. Across six QA tasks, DFM performs better over the strongest baseline by 12.71% on Mistral-7B-v0.3 and 11.19% on Qwen3-8B-Base. Beyond QA, domain adaptation yields average gains of 6.51 and 4.38 points over Qwen3-4B-Base and Qwen3-8B-Base, respectively.

Sun 18 OctComputation and Language
The gist
Large language models often struggle to remember or use facts not covered in their initial training. The authors propose Deep Fusion Memory (DFM), a method that predicts useful information from chunks of text to help the model remember better without slowing it down. This method merges predicted knowledge directly into the model's workings instead of just combining outputs, making it faster and requiring less data to train. Tests on question answering tasks and domain adaptation show this approach improves model performance significantly.
Open → 2610.25100v1

Algorithms predict better for annotators who agree with others

Who Do Perspectivist Models Predict Well? Annotator Agreement, Sparsity, and Performance

Abstract: Successful completion of perspectivist natural language processing (NLP) annotation tasks requires algorithms to be aligned across diverse perspectives in order to predict how different humans will rate text. This work studies how the performance of such algorithms varies for annotators who behave differently than others, either by providing annotations that diverge from others' responses or by providing fewer annotations altogether. To aid with characterizing model performance, we curate the largest collection to date of perspectivist NLP datasets with ordinal annotations. After evaluating multiple classes of perspectivist algorithms across each of the datasets, we model the relationship between individual annotator idiosyncrasy and algorithm performance. In general, we find that algorithms perform better on annotators who tend to agree with others, but that annotator response rate does not have a statistically significant positive correlation with algorithm performance.

Thu 15 OctComputation and Language
The gist
Some language tasks ask many people to rate the same text in their own way. The authors studied how well computer programs predict these human ratings, especially for people who often disagree with others or contribute fewer ratings. They found that algorithms guess better for people whose answers agree more with others. Surprisingly, how often a person rated did not make the algorithms more accurate.
Open → 2610.22567v1

Sparse autoencoders help understand language model features better

Sparse Autoencoders for Mechanistic Interpretability in NLP: A Critical Survey of Recoverability, Interpretability, and Utility

Abstract: Sparse autoencoders (SAEs) have emerged as a promising approach to mechanistic interpretability by decomposing language-model activations into sparse, potentially interpretable features. However, differences in design assumptions, evaluation objectives, and experimental settings complicate conclusions about their effectiveness. We present a critical survey of 138 studies on SAEs in natural language processing. Our contributions are fourfold: (1) we propose a task-grounded taxonomy connecting dictionary design, feature operations, and NLP functions; (2) we examine how sparsity allocation, geometry, data coverage, and inference affect feature recoverability; (3) we assess evidence for semantic interpretability, mechanistic faithfulness, and behavioral utility; and (4) we synthesize findings across multilingual processing, retrieval, reasoning, classification, and behavioral regulation, highlighting the roles of objectives, baselines, supervision, and intervention pipelines. Through comparative analyses of published results, we reconcile apparent disagreements, identify unresolved questions, and propose directions for more rigorous evaluation. An anonymous research register https://anonymous.4open.science/r/sae-research-register-C331/register.html accompanies the survey.

Wed 14 OctComputation and Language
The gist
Understanding how language models work inside is important but hard. This paper looks at a specific method, sparse autoencoders, which try to break down model signals into simple parts that people can understand. The authors reviewed many studies to see how well these methods work and how useful and accurate their explanations are. They organize the findings, point out disagreements, and suggest better ways to test and improve these methods.
Open → 2610.20139v1

OpenTWBench measures large language models on Taiwan knowledge

OpenTWBench (Open Taiwan Benchmark): Taiwan-Domain Knowledge Evaluation for Large Language Models, and How the Measurement Apparatus Shapes the Score

Abstract: We present OpenTWBench, an open evaluation suite for Taiwan-domain knowledge in large language models, built around two benchmarks. tw-exam-bench contains 255,854 four-option questions across 148 academic subjects, from fifteen years of Taiwan's national examinations published as open data by the Ministry of Examination; answer keys agree across 31,312 duplicated question groups at 99.91%, with all 27 disagreements accounted for. A reviewed second pass withdrew 2,180 items whose stems depend on a figure the text does not carry; all scores use the remaining 253,674. tw-formosa-bench contains 1,696 difficulty-curated questions about everyday and contemporary Taiwan - the Taiwan that licensing examinations never ask about - generated from a community-curated corpus under a discipline in which no model writes a ground truth, and curated against a frozen seven-model pool so that the pool scored at the passing margin (46.9-74.6%, median ~54%); a leave-one-out experiment shows the selection preserves model ranking (near-zero bias). Within six weeks the frontier had cleared that band: of 108 models now on the public board, 22 score above its ceiling, the best 91.6%. We also document how much of a score comes from the apparatus rather than the model. On identical questions, switching answer extraction from pattern matching to an explicit \boxed{} answer moves accuracy from 20.8% to 93.9%; a single placeholder character in a system prompt moved one model from last (58.2%) to first (83.9%) on the same benchmark; doubling the thinking budget from 8k to 16k tokens bought 0.2% while shuffling which questions failed; and a three-judge validation panel proved to be one judge in disguise, two members approving 92-99% of everything shown. We release all datasets, per-item difficulty data, curation manifests and a leaderboard at https://opentwbench.ai

Wed 14 OctComputation and Language
The gist
Measuring how well language AI models know Taiwan-related facts is tricky. The authors created OpenTWBench, a test suite with questions from Taiwan's national exams and everyday Taiwan topics. They show how the way answers are extracted or how the test is set up can dramatically change model scores. This new benchmark and its open data let others fairly compare AI understanding of Taiwan knowledge.
Open → 2610.19848v1

Evaluating reliability of large language models as judges in NLP tasks

All Verdicts are Not Equal: Rethinking LLM Judge Reliability

Abstract: LLM-as-a-Judge is the standard paradigm for NLP evaluation, yet its systemic reliability remains poorly understood despite being widely treated as a deterministic ground truth. We present a comprehensive reliability audit, stresstesting six frontier models across four benchmarks, five prompt formats, two presentation orders, three sampling temperatures, and ten repetitions per condition. Our empirical analysis reveals severe vulnerabilities: verdicts change across identical replications at temperature zero, position-order swaps flip the majority of verdicts on challenging tasks, and the most deterministic judge achieves perfect consistency by trivially repeating incorrect verdicts, agreeing with ground truth only 51% of the time. To formalize these multi-faceted failure modes, we introduce the trustworthy verdict rate (T ), a unified metric capturing the joint probability that an evaluation is reproducible, order-invariant, and accurate. UsingT , we derive a theoretical upper bound on accuracy imposed by position bias and show that reliability is item-specific rather than modellevel. Finally, we demonstrate that shifting from pairwise win-rate to holistic rubric scoring improves trustworthiness more than any single-format prompting intervention, offering an actionable framework for robust NLP evaluation.

Thu 8 OctComputation and LanguageArtificial Intelligence
The gist
Large language models (LLMs) are often used to judge the quality of language tasks, treated like reliable truth. This paper shows that these models can be inconsistent and sometimes give different or wrong answers depending on tiny changes. The authors create a new way to measure if judgments are trustworthy by checking if they are consistent, fair, and accurate at once. They find that evaluating by scoring full rubrics instead of simple win-or-lose choices makes the judgments more reliable.
Open → 2610.12083v1

Large language models recover from unwanted changes after forgetting

Not Every Change Is Necessary: Recoverable Drift in Large Language Model Unlearning

Abstract: Machine unlearning in large language models aims to remove unwanted knowledge while preserving the model's remaining capabilities. Although existing methods use retention objectives or restrict where edits occur, achieving the desired forgetting level can still leave collateral changes that impair non-target behavior. Our recovery comparisons suggest that some of these changes can be reversed while preserving observed forgetting performance. In this work, we present Propose-Then-Project Unlearning (PTP-U), a framework that combines targeted forgetting with the recovery of non-target capabilities. PTP-U first applies local analytic edits to weaken target knowledge associations, then aligns non-target output distributions with those of the original model to recover capabilities while maintaining fixed forgetting constraints. Both stages serve a common goal: satisfying the forgetting requirements while preserving fluent generation and performance on non-target tasks. Across three benchmarks, PTP-U achieves the strongest forgetting-retention trade-off among evaluated methods, reaching 81.22%-91.03% forgetting while preserving 94.20% non-target utility on average. At matched forgetting, PTP-U consistently retains higher non-target utility.

Thu 8 OctComputation and Language
The gist
When large language models forget specific information, they sometimes lose other abilities by mistake. The authors show that some of these accidental losses can be fixed without undoing the forgetting. They introduce a method called Propose-Then-Project Unlearning (PTP-U) that carefully removes targeted knowledge and then restores other skills. This method achieves better forgetting of unwanted information while keeping most other abilities intact compared to previous approaches.
Open → 2610.11915v1

Large language model bias testing improved with dual pairs across languages

The Missing Minimal Pair: Stereotype Evaluation in LLMs

Abstract: A common approach to measuring bias in Large Language Models is to compare the log-likelihoods of two contrastive stereotype sentences. We argue that such single-pair comparisons are often unreliable: simply rewriting the same stereotype with an alternative attribute can yield logically inconsistent preferences. To address this, we propose a dual minimal pair setup that introduces two axes of comparison for robust stereotype evaluation. First, we present a data-augmentation framework that fills critical gaps in existing stereotype datasets by generating paraphrases and alternate attributes. We apply our framework on a set of English, Russian, Spanish and Chinese stereotypes. Second, we introduce two evaluation metrics tailored to the dual minimal pair setup. One of these metrics provides a new perspective on bias by modeling the mutual information (MI) between social groups and stereotyped attributes. This MI-based metric is better suited for aggregation and enables more robust comparisons of stereotype strength across different languages and models. Our code is available at https://github.com/stepanat/missing-minimal-pair/.

Tue 6 OctComputation and Language
The gist
Measuring bias in large language models usually involves comparing two sentences that differ by a stereotype. The authors found that this method can be unreliable because small changes in wording can confuse the results. They created a method using two pairs of sentences with different stereotype attributes to get more reliable measurements. Their approach works in several languages and introduces a new way to score bias by looking at the relationship between groups and stereotypes. This helps compare biases more fairly across different models and languages.
Open → 2610.08747v1

Language models find facts by their order of mention in text

The Geometry of Contextual Relations: Language Models Address Facts by Order of Mention

Abstract: Human reasoning depends on how objects are related within propositions. \textit{How do relations organize the language representations of contextual contents?} We give an LLM a list of facts in its context (e.g., \emph{Alice eats an apple. Bob eats a pear.}) and measure how its hidden state changes when the question switches from what Alice eats to what Bob eats. Averaged over many lists, this change is a steering vector, which we call the \emph{ordinal vector}. It points to a fact by its \emph{order of mention}, the order in which the facts were stated in the context. We find that LLMs represent the fact a question asks about by its order of mention, not by the name the question contains. We state this as the \textit{ordinal addressing hypothesis}: each order of mention has a \emph{fact address} in the model's state, shared by all contexts, and a question moves the state to the fact address of the fact it asks about, while the context supplies what that fact says. Across Qwen, Gemma, and Llama, fact addresses are (1) \emph{ordered by mention}: query states are organized by the order of facts, not of names, even when one fact has multiple subjects; (2) \emph{steerable}: added to a question about the first fact of a new list, the ordinal vector makes the model answer with the second fact of that list; (3) \emph{low-rank}: they span a low-rank subspace in which the first-mentioned fact is the easiest to reach, surprisingly similar to human recall; and (4) \emph{emergent}: they are shared in late-middle layers, hold from 1.5B to 32B parameters, and form early in pretraining. Language models reach a stated fact by where it was mentioned, deepening our understanding of LLM reasoning.

Thu 1 OctComputation and LanguageArtificial Intelligence
The gist
This paper shows that large language models (LLMs) remember facts not by the names mentioned, but by the order in which they appear in the text. When asked about a fact, the model uses a kind of internal pointer that corresponds to the fact’s position in the list, rather than the specific words naming it. The authors call this the "ordinal addressing" idea, and they show it works across several popular LLMs and sizes. This discovery helps us understand how LLMs organize and recall information from their input context.
Open → 2610.00910v1

Latent information feedback improves transformer language models performance

Pretraining Latent Information Feedback Transformers with Teacher Supervision

Abstract: Transformer language models (LMs) are feed-forward: deep-layer representations are never fed back to shallower layers, and the only pathway for information to flow downward across generation steps is the decoded token. This narrow channel forces models to recompute intermediate results and to discard alternative continuations. In this work, we remove this bottleneck during pretraining, introducing the LIFT (Latent Information Feedback Transformer) architecture and training method which enable LMs to propagate state across generation. We achieve this by turning recurrent-state learning into a teacher-forced prediction problem: each input token is paired with an information-dense state, derived from the next-token distribution of an off-the-shelf pretrained LM. The model, extended with a small number of additional parameters, is then trained to predict both the next token and the next state. As the input states are precomputed, pretraining remains fully parallel across positions. At inference, the model's own predicted states are fed back, with a minor computational overhead that decreases with model size. Experiments with pretrained models ranging from 135M to 1B parameters show that LIFT consistently outperforms standard Transformers and baselines on language modeling, downstream reasoning tasks, and procedural tasks under token-matched budget, while being on par with or ahead of compute-matched Transformers. Moreover, a controlled study on a state-tracking task shows that a tiny LIFT outperforms same-size Transformers trained on 8x more data, even when trained with the states of a Transformer that fails the task. Overall, we show that LMs can learn to exploit deep-to-shallow feedback during pretraining via scalable teacher supervision.

Tue 29 SeptComputation and Language
The gist
Transformer language models usually process information in a one-way, feed-forward manner, which limits their ability to remember and use previous details efficiently. The authors propose a new model called LIFT that allows information from deeper layers to be reused in earlier layers during training. They do this by teaching the model to predict both the next word and extra helpful information about the upcoming words, using guidance from an existing language model. Tests show that LIFT models perform better on language understanding and reasoning tasks while also being more efficient during use.
Open → 2609.38149v1

Efficient attention method speeds up large context processing

SANTA++: Sampling Attention through Representative Keys

Abstract: Attention often concentrates on a small subset of tokens in the context, but which subset matters changes from one query to the next. To exploit this changing structure, we introduce SANTA++, a training-free stochastic attention method that uses representative keys for memory-efficient selection without scanning the entire key-value (KV) cache. Cached keys are organized into teams, and the query scores one representative from each team to decide which teams to sample. We compute exact attention scores within the sampled teams and reweight each team's contribution by the inverse of its inclusion probability. This importance sampling correction estimates attention over the full cache, with a sampling budget that lets us trade memory reads for accuracy. Remarkably, with 32 or 64 sampled teams, SANTA++ uses 16% to 22% of dense attention's KV reads and retains 94% to 99% of the dense-attention baseline's scores on LongBench v2 and HELMET's retrieval-augmented generation subset, and 85% to 91% on RULER, with Qwen2.5-7B-Instruct at 32K context. With 31 sampled teams, our GPU implementation delivers a $1.69\times$ attention speedup over the dense FlashAttention baseline at 32K context. By reducing the number of cache entries read, SANTA++ in principle complements architectures with compressed KV representations, such as multi-head latent attention. Our kernels are available at: https://github.com/OPUSLab/santapp-kernel-demo.git.

Mon 28 SeptMachine LearningComputation and Language
The gist
Processing long sequences of information in AI models can be slow because they have to look at everything at once. The authors introduce SANTA++, which cleverly picks smaller representative parts to pay attention to without having to check everything. This approach keeps results very close to full processing but reads much less data, making it faster and more efficient. Their tests show it works well on tasks requiring understanding of very long context.
Open → 2609.35629v1

Neural language models learn meanings from sentence structures not just words

Neural Language Models Learn the Contextual Distributions of Dependency Structures: a statistical learning theory to compositionality

Abstract: It is unclear how Neural Language Models (NLMs) acquire the structural meaning encoded by grammatical structures that is independent of lexical semantics. We propose a statistical learning process in which learned dependency structures themselves become new distributional units for subsequent statistical learning. Under this account, once a dependency structure is acquired, the model tracks its contextual distributions. These contextual features reflect the semantic properties of a composite structure. To test this hypothesis, we design a synthetic grammar in which each grammatical structure has distinct contextual distributions that cannot be recovered from the distributional statistics of their component tokens alone. We train a series of BERT-style masked language models on this grammar and examine their developmental trajectory. The results show that models can successfully learn the contextual distributions of composite dependency structures even though they cannot be inferred from token statistics alone. Developmental analysis further reveals a clear developmental trajectory. The learning of the dependency relations that define a grammatical structure consistently precedes the learning of its contextual features. These findings suggest that statistical learning in NLMs is not merely the accumulation of token co-occurrence statistics, but a process in which learned dependency structures become new units of distributional learning. We argue that this process provides a statistical-learning account of how NLMs solve the compositionality problem in language. Finally, we discuss the possibility that this statistical learning process provides an explanatory theory on how language cognition could emerge from pure distributional statistics.

Mon 28 SeptComputation and Language
The gist
Language models usually learn by looking at word statistics, but it’s unclear how they understand sentence structures that carry meaning beyond words. The authors show that these models can learn patterns of how grammatical structures appear in different contexts, even when those patterns can't be guessed by looking at individual words. They trained models on a made-up language where each structure had unique context patterns and found that models first learn the grammar relationships and then the meanings from context. This suggests that models build understanding by treating grammatical structures as new units for learning meaning.
Open → 2609.34936v1

Large language models unevenly recognize names by race and gender

Who Gets a Token, and What Does It Carry? Unequal Name Support and Concept Access in Large Language Models

Abstract: Names are personal identifiers, but they also carry social meaning and are widely used to evaluate how language models treat different people. Such evaluations typically assume that matched names are comparable model inputs. We show that this assumption often fails at the lexical interface: matched names are not necessarily matched inputs. Some names receive direct single-token access, while others are assembled from multiple subwords, creating unequal name-surface support. Across nearly half a million first names and 12 LLM-associated tokenizers, direct lexical access is highly selective, model dependent, and uneven across race- and gender-associated name metadata. We introduce NameTrace, a model-native, fine-grained, pre-behavioral framework for measuring whether unequal name-surface support remains a vocabulary property or becomes visible in task-relevant internal representations. NameTrace measures concept accessibility from the model's own probabilities over task-specific adjective axes with continuous task-aligned weights. On matched atomic and short-fragmented names within the same race/ethnicity--gender-associated strata, support predicts systematic differences in concept accessibility across fellowship, hiring, clinical assessment, and lending. These differences persist across all eight matched strata, extend across model families, and transfer to unseen names. Hidden-state interventions further show that the measured task directions have downstream leverage, shifting later constrained choices. Unequal lexical support is therefore demographically structured at the input and remains visible in task-relevant model computation. NameTrace makes lexical comparability measurable, supporting a broader principle: behavioral comparability begins with lexical comparability.

Mon 28 SeptComputation and LanguageArtificial IntelligenceComputers and Society
The gist
Different names aren't always treated equally by large language models because some names are recognized as single tokens, while others are broken into smaller parts. The authors show this unequal tokenization happens more often depending on the name's race or gender associations. They created a method called NameTrace to measure how this affects models’ understanding of concepts linked to those names. Results indicate that these differences matter for tasks like hiring or lending decisions made by the models.
Open → 2609.34065v1

Language models improve learning from context using perturbed documents

Learning to Learn from Context: Synthetic Training from Perturbed Public Documents

Abstract: Real-world tasks often require large language models (LLMs) to learn from complex task-specific context rather than pretrained parametric knowledge. This capability remains a weakness of LLMs, while human annotation for such task contexts is expensive and difficult to scale. Public high-quality documents are an abundant alternative, but much of the public web has already been consumed during pretraining: training on such documents naively would reward memorization rather than context learning. In this work, we attempt to make use of high-quality public documents with small perturbations and empirically find that LLMs can successfully generate context-dependent reasoning traces and answers, which are then used to train a student model. Specifically, we construct a synthesis pipeline that (i) rewrites source documents to reduce memorization risk, (ii) generates questions and rubrics that require reasoning over the document, (iii) answers the questions with the document as context, and (iv) admits only samples that genuinely depend on the document. Without any human annotators, our pipeline generates about 10k samples from 3.5k documents, and the resulting student model substantially improves the performance on CL-bench. SFT raises a Qwen3.6-35B-A3B student from 13.7% to 22.8%, and a subsequent rubric-reward RL stage reaches 24.6%, on CL-bench comparable with a frontier model of over a trillion parameters, Qwen3.8-2.4T (23.9%). We also observe a broad transfer of improvements to long-context understanding, instruction following, and reasoning, while code generation and knowledge remain mostly flat. We hope this work provides a reproducible and scalable way to improve the ability of LLMs to learn from context, and to facilitate further research on context-grounded reasoning.

Sun 27 SeptComputation and LanguageArtificial Intelligence
The gist
Large language models often struggle to understand and reason with new information provided in a task's context. The authors demonstrate a method that alters public documents slightly to prevent models from just memorizing them and instead encourages true understanding. Using this approach, they generate synthetic training data without needing expensive human labeling. Training smaller models on this data helps them better use context for reasoning, improving their performance on certain benchmark tests.
Open → 2609.33642v1

Researchers map detailed internal features for time aware recall in language models

Identifying Temporal Features within Transcoders for Time Sensitive Factual Recall

Abstract: Large Language Models (LLMs) suffer from temporal misalignment, often due to the contradictory nature of their training corpora. While current mitigation strategies rely on computationally expensive fine-tuning or context-heavy retrieval augmented generation (RAG), the internal mechanisms governing time-sensitive recall remain under-explored. Unlike prior studies that identify temporal components such as attention heads and MLP layers, we provide the first feature-level map of temporal recall by isolating individual MLP features via transcoder circuit tracing. We identify three node categories (common temporal, common to the year, and chrono-semantic) which interact to generate a temporal filter during factual recall. By analysing Gemma 2 2B, LLaMA 3.2 1B, and Qwen3-4B, we show that these features do not follow a simple linear pipeline but represent time through a parallel and mixed syntactic-semantic interplay across layers. We additionally discover a class of higher-layer temporal components invisible to existing EAP-IG methods, establishing transcoders as a more complete lens for temporal interpretability in time-sensitive factual recall. These findings present MLP components for potential targeted interventions in time-sensitive factual recall

Sun 27 SeptComputation and LanguageMachine Learning
The gist
Language models sometimes get facts wrong because they mix up info from different times. The authors found specific tiny parts inside these models that help them remember facts correctly based on time. They looked at different models and saw that remembering time happens in a complicated way, not just step by step. Their work can help improve how language models keep up with facts that change over time.
Open → 2609.33183v1

Large language models improve reasoning with semantic abstraction framework

Semantic Abstraction for Natural Language Inference: a Methodological Framework for Discovering and Compensating Semantic Knowledge and Reasoning Gaps in Large Language Models

Abstract: Despite their outstanding performance on many NLP tasks, LLMs face serious challenges related to semantic abstraction. In this study, we are interested in understanding how LLMs leverage abstract semantic knowledge in natural language inference (NLI), which requires sophisticated linguistic capabilities to interpret implicit meanings, contextual conceptual relationships, and semantic connections between words and phrases. To this end, we propose a methodological framework for constructing new semantic knowledge at a higher level of abstraction, which we define under the notions of semantic compatibility and incompatibility for NLI. In this framework, the meaning of the lexical-semantic relations between the premise and the hypothesis is reconfigured to achieve a more flexible semantic network that induces different reasoning paths in LLMs. These new pathways show a consistent pattern of responses that allows agreement on a single response. The results demonstrate that our proposal allows to discover and compensate for LLMs' semantic knowledge gaps in NLI, achieving significant improvements in accuracy, exceeding 10% for some models, and in particular for the non-entailment class. It is essential to note that LLMs need structured knowledge and not just more data to bridge reasoning gaps. Our hybrid approach directs attention to overlooked word relationships, allowing models to synthesize missing information. We believe that the future lies not in increasing model size, but in creating a semantic scafolding that mimics the flexibility of human thinking. Hopefully, our proposal will enable the development of more robust agents and interpretable reasoning, guiding AI toward reliable language understanding.

Tue 22 SeptComputation and Language
The gist
Large language models (LLMs) sometimes struggle to understand deeper meanings between sentences during tasks like language inference. The authors propose a framework that reorganizes word relationships to help these models reason more like humans by discovering missing knowledge. Their approach boosts model accuracy by over 10%, especially in understanding when statements do not follow from each other. This method focuses on structured knowledge rather than just adding more data, helping LLMs fill in gaps in their reasoning.
Open → 2609.26610v1

Culture language and region annotations enhance web data benchmarking

FineWeb-CLaR: Culture, Language, and Region Annotations for Benchmark-Aligned Corpus Auditing

Abstract: Cultural evaluation coverage and robustness in language models are difficult to diagnose because pretraining corpora and cultural benchmarks are rarely indexed with comparable metadata. Benchmarks increasingly target culturally situated phenomena at the level of languages, regions, and locale-specific practices, while web-scale corpora are usually organized only by language. A shared culture-language-region layer makes these resources comparable, enabling audits of whether a target cultural phenomenon is represented in pretraining data, evaluated by benchmarks or both. To this end, we introduce FineWeb-CLaR, a large-scale annotated dataset derived from FineWeb and FineWeb-2 that places web documents on a shared culture-language-region axis for corpus auditing and benchmark alignment. FineWeb-CLaR annotates the full 30.9B-document collection from FineWeb and FineWeb-2 with URL-derived region labels and cultural-topic provenance. Our region resolver assigns a non-empty region to 25.61% of documents (7.92B). For cultural-topic analysis, we induce locale-specific topics and project them onto the 14 leaves of the Cultural Taxonomy of Liu et al. (2025), producing Locale Topic Distributions (LTDs) for corpus-side comparison. We also annotate 277 cultural NLP benchmarks with the same taxonomy, language coverage, and region coverage. Together, these resources enable direct comparison between corpus-side pretraining evidence and benchmark-side evaluation coverage.

Mon 21 SeptComputation and Language
The gist
It’s hard to tell if language AI models understand different cultures because the training data and tests are labeled differently. The authors created FineWeb-CLaR, a big dataset that tags web documents with culture, language, and region information, making it easier to compare what’s in the training data with what tests measure. They also labeled many benchmarks to match this system, so it’s clearer if models truly cover cultural topics. This helps check if AI learns from or is tested on diverse cultural content.
Open → 2609.25298v1

Large language models show varied political leanings based on test setup

Navigating the digital spectrum: Assessing political bias, stability, and downstream fairness in Large Language Models

Abstract: Large Language Models are increasingly deployed as information intermediaries, yet measuring their political behavior remains fragile because questionnaire results mix model dispositions with measurement artifacts and response-elicitation biases. We introduce a robust Political Compass Test evaluation framework that samples 300 configurations across an eight-dimensional perturbation space varying language, framing, instructions, answer format, option order, and persona wording. We evaluate eight Gemma 3 and Qwen 3 models across 14 languages and three quantization levels, obtaining design-averaged political coordinates with quantified uncertainty. Most models lean Libertarian-Left on average, but instruction phrasing, language, and answer format significantly affect recovered coordinates. Cross-lingual differences primarily reflect coordinate drift rather than distinct cultural reasoning. Reverse-engineering the test also exposes axis-weighting imbalances and the collapse of degenerate responses toward the center, so near-origin estimates for the smallest models can reflect weak signal rather than centrism. Free-text reasoning and chat-then-classify elicitation alter recovered coordinates, and larger models show clearer persona separation, with a specific failure of the Authoritarian-Left persona to move most models in the intended social direction. In downstream tasks, persona effects are modest relative to model size and target group for hate-speech detection, while base and centrist prompts give the highest agreement for topic-level sentiment. Political role prompting therefore has measurable but task- and dataset-specific downstream effects.

Tue 8 SeptComputation and LanguageComputers and Society
The gist
Large language models are used to provide information, but it's tricky to know their political leanings because test results can be affected by how questions are asked and other factors. The authors created a detailed evaluation method that changes many test settings to see how these affect the models' political scores. They found that most models lean toward the Libertarian-Left, but changes in language, instructions, and answer formats can shift this. Also, smaller models can appear neutral simply because their answers are weak signals instead of truly centrist views.
Open → 2609.08637v1