Papers for

llm developers

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

Large language models uncertainty measured with two level ensembles

Black-Box Uncertainty Quantification for Large Language Models via Ensemble-of-Ensembles

Abstract: Reliable use of large language models (LLMs) requires uncertainty estimates that separate ambiguity intrinsic to a prompt from genuine knowledge gaps. Bayesian and deep-ensemble methods deliver this aleatoric/epistemic decomposition in principle, but are computationally prohibitive at LLM scale and require white-box access. Existing black-box methods, in contrast, summarize to a single scalar that cannot make the distinction. We introduce a two-level ensemble that bridges this gap. Stochastic decoding of a fixed prompt forms the inner ensemble; meaning-preserving semantic perturbations of the prompt form the outer ensemble. Variability is measured in a continuous embedding space, and the law of total covariance gives an exact within-/between-perturbation decomposition to estimate aleatoric and epistemic uncertainty (AU/EU); it requires only black-box sampling access. On top of this estimator, we contribute three theoretical results that ground the framework: (i) a finite-sample bias identity for nested Monte Carlo with a closed-form bias-corrected epistemic estimator; (ii) a Bayesian bridge bounding the gap between perturbation-population AU/EU and Bayesian targets; and (iii) a separation result identifying a class of failure modes that any consistency-only black-box method provably misses while perturbation-based EU detects. Across five short QA benchmarks and four instruction-tuned LLMs spanning two families and two scales, our estimator is competitive with strong white-box baselines, and at a matched generation budget it matches leading black-box methods on error ranking while decomposing AU and EU; controlled paired interventions show that AU and EU respond selectively to distinct, actionable sources of failure: prompt ambiguity and model knowledge gaps; and a long-form pilot on the NQ long-answer split shows the framework extends to claim-level factuality without retraining.

Fri 13 NovMachine LearningArtificial Intelligence
The gist
It is hard to tell if a large language model is unsure because a question is unclear or because it doesn’t have the knowledge. The authors created a way to measure these two types of uncertainty separately by looking at different model answers and slightly changed versions of the question. This method only needs to observe the model’s outputs, without accessing its internal details. Tests on various question-answer tasks show the new method works well and can identify specific reasons for mistakes.
Open → 2611.18453v1

Refusal removal does not get uniformly harder in newer large language models

Is Abliteration Getting Harder? Not Uniformly: Refusal Removal by Direction-Based Ablation and ARA in Heretic on the Qwen3.8, Gemma 4, Olmo 3.1 and Mistral Small 3.2 LLMs Versus Their Predecessors

Abstract: Abliteration, a fine-tuning-free weight edit, removes refusal from open-weight large language models (LLMs). The developer of Arbitrary-Rank Ablation (ARA), an optimisation-based variant, reports that direction-based abliteration, which ablates a refusal direction, increasingly fails on newer models. On five release pairs, refusal removal did not become uniformly harder. Qwen2.5-32B and Qwen3.8-27B, Gemma-2-27B and Gemma-4-31B, OLMo-2-32B and Olmo-3.1-32B, Mistral-Small-2409 and Mistral-Small-3.2, and Qwen2.5-7B and Qwen3.5-4B were edited in Heretic 2.0 by three or four direction-based methods (difference-in-means, MPOA, SOM and, on the small pair, J-space) and ARA under the same prompts, nominal search budget, first-token KL divergence cap and selection rule; removal counted as harder when the automatic WildGuard judge found more of 972 held-out harmful prompts (JailbreakBench, XSTest, StrongREJECT, HarmBench and an author-compiled malicious-code bank) still refused. After direction-based removal, newer releases retained -1 to +37 percentage points more refusal, and after ARA -26 to +34. With run-to-run variance included, between-pair heterogeneity of the direction-based effect ranged from high to low with the scale and variance model (I^2 20-80%). ARA lost less effectiveness than direction-based methods in Gemma and OLMo and more in Qwen; each verdict rests on one ARA run, and some changed with the divergence budget or variance model. In Gemma the direction-based increase came from the malicious-code bank and reversed on public sets. Contrary to prediction, keyword counts at the tool's window did not undercount judged refusal on average, but differed by a mean 7 points per model. Audits of a single pair, method or prompt source can reach opposite conclusions. Whether a newer model resisted removal depended on the release pair, method and budget, not on generation alone.

Fri 13 NovArtificial IntelligenceComputation and Language
The gist
Large language models (LLMs) sometimes refuse to follow certain prompts for safety. A way to fix this without retraining, called abliteration, edits the model's weights to remove these refusals. The authors found that newer LLM versions are not always harder to fix this way—sometimes removal works better, sometimes worse, depending on the model, method, and prompt set. They also found variability depending on how the removal is tested, suggesting no simple pattern of increased resistance to refusal removal in newer models.
Open → 2611.17846v1

Jailbound framework helps test large language model safety limits

JailBound: FOL-Guided Jailbreak Evaluation for Exploring LLM Behavioral Boundaries

Abstract: As large language models (LLMs) are increasingly deployed as general-purpose conversational assistants, they must handle diverse user requests while reliably refusing harmful ones. Jailbreak attacks craft adversarial instructions or conversational contexts to elicit unsafe responses despite these safeguards. However, existing jailbreak benchmarks often rely on manually designed or collected prompts, making it difficult to systematically extend coverage to new threat conditions. Moreover, existing attack search methods typically optimize for attack success rate (ASR), providing limited insight into how model behavior changes locally between safe refusal and unsafe response. To address these limitations, we present JailBound, a unified jailbreak evaluation framework that combines automated benchmark construction with first-order loss (FOL)-guided dual-branch search. JailBound organizes threats by risk category, application domain, and attack type. Conditioned on this taxonomy, a fine-tuned meta-attack generator synthesizes evaluation prompts across diverse threat conditions. FOL serves as a local sensitivity signal for exploring high-value attack states and behavior-sensitive states. We evaluate JailBound across 46 LLMs from 13 model families under a unified experimental protocol. Under matched settings, the High-Value Branch improves average ASR over a ZOL-only baseline by 5.1 and 4.9 percentage points across the two evaluation settings. Under local perturbations, states from the Safety-Sensitivity Branch exhibit higher behavioral flip rates than those from the High-Value Branch across all tested radii for JailBound and S-Eval, indicating greater local sensitivity of judge-assigned safety labels. Warning: this paper includes examples that may be offensive or harmful.

Thu 5 NovCryptography and Security
The gist
Large language models need to avoid giving harmful replies even when users try tricky questions. The authors designed JailBound, a tool that creates lots of test questions automatically to find where models might slip up in safety. It looks closely at how models change from safe to unsafe answers and tests many big models with this method. This helps understand and improve the models’ safety against tricky user prompts.
Open → 2611.07530v1

COMPREHEND improves detailed prompt understanding for large language models

COMPREHEND: Ultra-Fine-Grained Benchmark Understanding by Entailment Hierarchy

Abstract: Understanding what kind of prompts users typically ask and how well an LLM performs on these prompt categories is crucial for LLM practitioners. The existing methods satisfy the demand by hierarchically clustering the user prompts and synthesizing the cluster names using an LLM to build a taxonomy. However, the cluster names from these methods are usually too general for humans to understand the detailed prompt distributions and identify the exact strengths and weaknesses of LLMs. In this study, we propose COMPREHEND, which automatically constructs an ultra-fine-grained entailment hierarchy for the prompts. The entailment hierarchy replaces the cluster names in the taxonomy with sentences to provide more precise cluster descriptions for more fine-grained categorization. Compared with EvalTree, a state-of-the-art taxonomy construction method, COMPREHEND improves an entailment/faithfulness metric from 22.6 to 41.7 and an informative metric from 16.3 to 39.1 in WildBench. Our experiments also show that the weakness report generated based on our entailment hierarchy is much more informative than EvalTree in terms of predicting the response scores in the benchmark. All the code is released in https://github.com/JackNapier20/COMPREHEND_public.

Tue 3 NovComputation and Language
The gist
Understanding the types of questions people ask large language models (LLMs) and how well these models handle them is important but challenging. The authors propose COMPREHEND, a method that organizes prompts into very detailed categories with clear sentence descriptions. This helps users see exactly where the model performs well or poorly, beyond broad groupings. Their method showed better accuracy and informative descriptions compared to previous approaches. COMPREHEND also produces clearer reports that help predict model performance on different prompt types.
Open → 2611.04249v1

Effect-aware evaluation reveals flaws in lLM multi-turn function calling

CAREFUL-Bench: Effect-Aware, Judge-Free Evaluation of Multi-Turn LLM Function Calling

Abstract: The multi-turn checker of the Berkeley Function Calling Leaderboard (BFCL) evaluates final state correctness but ignores the effects of intermediate calls: extra calls are free, function documentation does not describe side effects, and the scripted user never responds to proposals. Consequently, models asking for confirmation before destructive actions are penalized; 260 of 734 reference turns in the base category perform destructive actions without confirmation. We make BFCL effect-aware without replacing it. We infer effect declarations for all 128 functions by executing them in BFCL simulators and comparing state changes, with destructive writes identified by three annotators. A judge-free replay scorer reproduces official results across all 800 reference trajectories while reporting unrequested destructive calls, duplicate writes, redundant reads, surplus steps, and their token costs. We also introduce a confirmation-aware protocol with a deterministic simulated user. Across 11 open models, effect-aware scoring preserves the official ranking but reveals substantial differences in execution quality: the leading model loses 14.0 points due to unrequested destructive calls, while surplus steps consume up to 28% of input tokens. Adding effect hints to function documentation has no impact on eight evaluated models. Allowing the simulated user to respond recovers up to four points for models that seek confirmation. However, the best model completes only six of 576 conversations requiring destructive actions with explicit confirmation. These results expose a key limitation of current function-calling benchmarks: they cannot distinguish between asking for clarification and obtaining permission before potentially destructive actions.

Mon 2 NovArtificial IntelligenceSoftware Engineering
The gist
Current evaluations of large language models (LLMs) that call external functions often miss when models perform unexpected or harmful actions during conversations. The authors improved an existing evaluation by tracking the effects of every function call, especially destructive ones, without replacing the original system. They found that many models perform destructive actions without asking for confirmation and that scoring based on these effects changes how model behavior is judged. This work shows the limits of current benchmarks in distinguishing careful clarification from proper permission-seeking during critical actions.
Open → 2611.02978v1

AdKnob controls ad prominence and labels ads precisely in AI chats

AdKnob: Ad Intensity Control and Labeling for LLM-Native Advertising

Abstract: Large language model (LLM)-native advertising has emerged as a viable monetization channel as LLM service providers seek sustainable revenue streams. However, integrating native ads into LLM responses introduces two largely understudied challenges: (1) controlling ad intensity -- how prominently advertising content appears -- and (2) ad labeling -- identifying which segments of the generated text constitute advertising. We propose AdKnob, a framework jointly addressing both challenges. For ad intensity control, we introduce new ad control tokens without pretrained semantic associations and align the model via direct preference optimization to generate ads at each desired level. For ad labeling, we propose an attention rollout-based attribution mechanism that identifies ad segments by tracing their attribution to input ad-related tokens, requiring no additional inference cost. We further construct MI-KnobSet, the first multi-level ad intensity preference dataset for LLM-native ads. Experiments across six LLMs and human evaluations show that AdKnob achieves uniformly spaced ad intensities where baselines collapse to a narrow range, achieves the highest ad labeling accuracy while being over 500x faster than the strongest baseline, and is preferred over baselines by human evaluators up to 89.6% of the time.

Sat 31 OctArtificial IntelligenceComputers and SocietyMachine Learning
The gist
When chatbots powered by large language models show ads in their answers, it can be tricky to control how obvious those ads are and to clearly mark which parts are ads. The authors made AdKnob, a system that lets these models dial the amount of advertising up or down and label the ad parts exactly. They created new special words to help the model control ad levels and used a new way to trace ad content without slowing things down. Tests with several language models showed AdKnob works better and faster than past methods.
Open → 2611.00822v1

RoleFUZZER tests large language models for role adherence errors

RoleFUZZER: User-Behaviour Fuzzing for Role-Specification Adherence in Large Language Models

Abstract: Large language models (LLMs) are often conditioned into specific roles such as tutors, interviewers, or agents acting for an organisation, through system prompts that specify how they should behave. The extent to which they can actually adhere to these role specifications under multi-turn interactions with realistic and potentially problematic users is understudied. To address this, we propose user behaviour fuzzing, a guided search over plausible user behaviours that can lead a model to violate its role specification, and introduce RoleFUZZER, a framework that adapts coverage-guided fuzzing from software testing to operationalise the search. RoleFUZZER constructs scenarios guided by a user-behaviour coverage grid that crosses a grounded taxonomy of user interaction types and the role specification of the model under test i.e. what is under pressure. With RoleFUZZER, we evaluate LLMs across five role specifications, including tutor, financial advisor, AI interviewer, antagonistic roleplay character, and web agent under organisational policies. Across simulated and validated conversations in 15 campaigns, the framework reaches >= 89% coverage of the 64-cell behaviour grid for every model and role tested, and surfaces specification-grounded failure patterns. At an equal budget of conversations, removing the RoleFUZZER grid and asking an LLM to generate specification targeting users free-form lowers coverage to 39-69% with fewer kinds of violations. The same specification can also fail in opposite directions. For e.g., on the antagonist, open-weight target models override the player's safety signals, while Claude Haiku 4.5 breaks character instead. RoleFUZZER offers a tractable way to uncover vulnerabilities in LLMs across role specifications by testing diverse user behaviours that are likely to violate them.

Wed 28 OctComputation and LanguageArtificial IntelligenceSoftware Engineering
The gist
Large language models can act as tutors, advisors, or characters by following instructions on how to behave. However, it's unclear how well they stick to these roles when talking with different users over many turns. The authors created RoleFUZZER, a tool that tries lots of realistic user behaviors to find when these models fail to keep up their assigned roles. It uses a smart way to cover many types of user actions and shows where models make mistakes. This helps improve trustworthiness by discovering weaknesses in how models follow their roles.
Open → 2610.38438v1

Large language model failures revealed through internal reasoning states

Uncovering Why LLMs Fail: From Internal Mechanisms to Adversarial Reasoning Trajectories

Abstract: Large language models (LLMs) are usually evaluated by final-output correctness or safety, yet an endpoint label does not reveal how a failure develops throughout the reasoning process. We therefore ask whether ordinary reasoning failures and adversarially induced failures can be characterized using the same internal coordinates. Towards this end, we in this paper study a white-box diagnostic framework CauTrace to disclose the root-cause of why LLM fails from three fundamental inspections: Functional States, Propagation of Representation, Temporal-State Coordination and Reasoning Trajectory. First, seven functional states are creatively defined to represent the support and constraint processing in LLMs. Second, the propagation of representation change measures how the changes among the seven states persist across subsequent Transformer blocks. Third, temporal-state coordination and trajectory topology characterize when the commitment and verification occur relative to the support, and how the dominant state sequence evolves over the full inference process. Finally, we evaluate CauTrace on both clean reasoning failures and paired adversarial perturbations. Across Qwen and LLaMA-family models, the failed traces show weaker Attention-MLP agreement on support-state updates and less verification activity relative to commitment, while the exact representation-level signatures and timing shifts vary by architecture. Held-out repair and steering tests demonstrate consistent operational signals that were in line with our claims for certain models and specific settings, with aligned steering producing the predicted directional changes in process measures. Overall, CauTrace provides a unified diagnostic framework for analyzing failure dynamics through changes in support, commitment, and verification, which would shed light on the in-depth research on LLM security.

Tue 27 OctArtificial Intelligence
The gist
Large language models (LLMs) sometimes give wrong answers, but just seeing the final error doesn’t explain why they fail. The authors created a tool called CauTrace to look inside these models and track how their reasoning fails step by step. By defining different mental 'states' and measuring how the models change during reasoning, they found patterns where models struggle, especially with verifying information. Their approach helps explain failures in both normal and tricky test cases, offering clues for improving LLM safety and reliability.
Open → 2610.37369v1

Large language models tested for fairness with multiple identity traits

Beyond Single-Attribute Auditing: A Configurable Multi-Attribute Benchmark for Intersectional Fairness in LLMs

Abstract: Large language models (LLMs) are increasingly evaluated for fairness, yet most existing benchmarks audit one sensitive attribute at a time. This single-attribute paradigm can miss intersectional harms, where multiple attributes interact non-additively and produce failures that are invisible under marginal evaluation. We introduce ComFair, a configurable benchmark that converts existing fairness data into 479,066 controlled Dual- and Multi-attribute instances and pairs every augmented instance with its task-equivalent source. Across Recognition, Selection, Continuation, and Conversation, this pairing supports a unified audit of performance or bias-score shifts, how often added attributes introduce new errors or mask existing ones, and model-ranking stability. Experiments on ten LLMs show task- and subgroup-dependent sensitivity: discriminative tasks expose both transition directions even when aggregate changes are modest, generative tasks exhibit larger and directionally asymmetric shifts, and rankings vary with attribute composition. ComFair therefore provides a scalable controlled sensitivity audit---rather than a model of the natural joint distribution of identities---for testing the robustness of fairness-benchmark conclusions beyond single-attribute settings.

Mon 26 OctArtificial Intelligence
The gist
Evaluating whether language models are fair often looks at one sensitive trait at a time, like gender or race, which can miss problems that happen when traits overlap. The authors created ComFair, a big set of test cases that combine two or more traits and pair them with similar tasks without added traits, to see how this changes model behavior. They tested ten language models and found that some errors appear only when multiple traits are considered, and the severity and type of errors depend on the kind of task and attribute combination. This work helps check if fairness assessments that consider only one trait at a time might be missing important problems.
Open → 2610.35069v1

Benchmark measures long-term learning improvement in language models

Continual Learning Exam: Benchmarking Long-Horizon RSI and Experiential Learning in LLM Agents

Abstract: Continual learning is the ability of LLM agents to improve over time by acquiring, refining, and applying knowledge from experience while retaining useful prior knowledge. It is essential for agents learning over long time horizons, yet current evaluations largely assess individual task performance rather than improvement through accumulated experience. We introduce Continual Learning Exam (CLE), a large-scale benchmark developed by 50+ domain experts to measure experience-driven improvement in realistic professional workflows. CLE comprises 120 human-authored tasks, each organised as a five-level ladder, yielding 600 levels spanning 89 professional roles across 12 occupational domains. Agents must pass each level to unlock the next, with later levels building on experience accumulated during earlier assignments. An evaluation agent assesses each submission against human-authored rubrics. We compare a single-submission setting with one that provides diagnostic feedback after failure and allows one revision. Across 17 harness-model configurations using frontier LLMs, feedback and revision yield a median completion gain of 5.3 percentage points. Yet the best-performing system clears only 47.7% of levels and completes all five levels in just 23.3% of tasks, highlighting the difficulty of sustained learning from experience for existing frontier LLMs. As such, the dataset can also be used to post-train LLMs to strengthen their continual learning capabilities.

Sun 25 OctMachine LearningArtificial IntelligenceComputation and Language
The gist
Continuing to learn and get better over time is important for AI language agents, especially for complex professional tasks. The authors created a large test called Continual Learning Exam (CLE) to see how well these agents improve by gaining experience on multi-step workflows. The test includes 600 levels across various jobs and subjects, requiring agents to build on previous knowledge to solve harder tasks. They found that giving feedback helps agents improve, but even the best models struggle to fully master the tasks, showing there's still a lot of room to improve AI's long-term learning. This benchmark can also be used to train models to get better at learning over time.
Open → 2610.33956v1

LLM memory errors reduced with counterfactual scenario testing

ConMem: Counterfactual Multi-Scenario Probing for Resolving LLM Memory Misalignment

Abstract: Long-term memory is critical for autonomous Large Language Model (LLM) agents executing multi-session, long-horizon tasks. However, existing memories often inappropriately store, retain, or retrieve over-generalized, misattributed, or privacy-sensitive information, a problem we term memory misalignment. To study this problem, we construct \textbf{MemAlign}, a benchmark comprising 1,079 curated memory cases grounded in synthetic, validated multi-session conversational contexts covering diverse misalignment taxonomies. We then propose ConMem, an algorithm that reframes retention decisions as prospective counterfactual causal inference. \textbf{ConMem} synthesizes counterfactual interaction scenarios to prospectively audit downstream intervention effects across three aspects: relevant task execution, irrelevant invariance checks, and risk-sensitive boundary shifts. Experiments show that ConMem outperforms other methods, achieving 75.16\% accuracy, 64.15\% balanced accuracy, and a Macro-F1 of 65.31\%, effectively mitigating over-retention biases without sacrificing personalization utility.

Sat 24 OctArtificial Intelligence
The gist
Large language models sometimes remember things badly, mixing up details or keeping too much private information. The authors create a dataset to study this problem of 'memory misalignment.' They introduce a method called ConMem, which imagines different scenarios to check if remembering certain things helps or hurts future tasks. Their method improves accuracy in managing memory, keeping useful personalization while avoiding mistakes.
Open → 2610.33432v1

Post-training methods affect large language models value preferences differently

How Does Post-Training Shape LLM Value Preferences?

Abstract: During post-training, large language models (LLMs) learn value preferences from corpora that may contain imbalanced proportions of opinions associated with different demographic groups. Amplifying these imbalances may further underrepresent some groups' opinions. However, how opinion mixtures shape value preferences across post-training paths remains unclear. To answer this question, we conduct controlled experiments using post-training corpora with different opinion mixtures and five post-training paths based on SFT, DPO, and GRPO. We find that post-training amplifies opinion imbalances differently across post-training paths. Once value preferences are amplified, whether further post-training can remove them depends on both the post-training method and the opinion mixture in the corpus. More concerningly, a model's stability under interventions depends on whether further post-training changes its value preference. Moreover, these value preferences are transferred to student models even when distillation uses corpora with no explicit opinion questions. These findings suggest that both the opinion mixture in post-training corpora and the choice of post-training method should be considered when developing, evaluating, and reusing LLMs. Our code and data are available at https://github.com/ku-nlp/llm-value-preferences.

Fri 23 OctComputation and Language
The gist
Large language models (LLMs) learn from text that contains opinions from many groups. The balance of these opinions can shape the models' values in different ways during a stage called post-training. The authors found that different post-training methods can increase or decrease these opinion biases differently. They also discovered that once certain values are learned, it's not always easy to change them later, and these values get passed on when making smaller, student models. This means that both the choice of training data and the method matter for how these AI systems develop their value preferences.
Open → 2610.33371v1

Contrastive decoding improves reasoning reliability in large language models

Mitigating Query Shortcuts in Chain-of-Thought with Context-Aware Decoding

Abstract: While Chain-of-Thought (CoT) prompting improves the reasoning performance of large language models (LLMs), recent studies have shown that final answers do not always depend on the generated rationales. In particular, LLMs may rely on query-specific shortcuts rather than the information provided by their reasoning steps. In this paper, we investigate whether context-aware decoding (CAD), an existing contrastive decoding method, can mitigate such shortcut behavior when applied to CoT reasoning. We study this application as Rationale Contrastive Decoding (RCD), which contrasts the next-token distributions conditioned on the query with and without its rationale, amplifying tokens whose likelihood increases due to the rationale. We provide a theoretical interpretation of this decoding rule via a KL-regularized reward-maximization objective. Experiments across five reasoning benchmarks demonstrate that RCD improves answer accuracy over standard CoT. Furthermore, intervention analyses using ground-truth and random rationales reveal that RCD increases the behavioral dependence of final answers on the rationale content. Our findings suggest that contrastive decoding offers a simple, training-free approach to mitigating query shortcuts.

Wed 21 OctComputation and LanguageArtificial IntelligenceMachine Learning
The gist
Sometimes, large language models give the right answers without actually using the detailed reasoning steps they explain. This means they might just take shortcuts based on the question instead of truly thinking through the problem. The authors studied a new way called rationale contrastive decoding, which helps the model rely more on its reasoning steps when deciding on answers. Their tests show this method improves accuracy and makes answers depend more on the reasoning provided. This approach works without needing to train the model further.
Open → 2610.29518v1

Large language models balance what they say and what they know better

Speak vs. Know: Balancing Behavior and Knowledge Distributions of Large Language Models

Abstract: Recent work reports that the distribution verbalized directly in the output text aligns far better with human opinion distributions than the next-token log-probability distribution of LLMs. However, the verbalized distribution reflects what LLMs know about a group's opinions (knowledge distribution), while what they actually output at deployment is governed by their log-probability distribution (behavior distribution). We find that when prompting LLMs to predict the opinion of a specific group, the most likely answer under the verbalized distribution differs from that under the log-probability distribution for 37% of World Values Survey questions, averaged over three LLMs. Thus, LLMs' knowledge does not match their behavior. This matters for evaluation: a verbalized distribution can score well against a group while its most likely answer is one the model never outputs, so the score no longer describes the model. The alignment scores for these two distributions reflect only the model's behavior or its knowledge, separately. We therefore propose a learnable combination of the two distributions that follows the most likely answer of the behavior distribution while preserving the shape of the knowledge distribution, with a single hyperparameter trading off behavioral fidelity against distribution quality. Our experiments on three opinion survey datasets and six models show that the balanced distribution improves behavioral consistency over the verbalized distribution from 71.6% to 95.6% on average, while the total variation distance to the human opinion distributions rises only slightly from 28.8% to 33.8%, and it remains effective under temperature scaling and cross-dataset settings.

Tue 20 OctComputation and Language
The gist
Large language models (LLMs) sometimes say different things than the knowledge they have about people’s opinions. The authors show that the answers LLMs most often produce don’t always match what the models ‘know’ about public opinion. They suggest a new method that mixes the model’s output choices with its knowledge to make answers more consistent while still reflecting real opinions. This approach makes the models’ behavior better align with what they understand about group opinions.
Open → 2610.27648v1

Llms show weaknesses under adversarial cultural value tests in chinese

ABench: Adversarily Benchmarking Large Language Models across Taxonomized Values

Abstract: Ensuring the alignment of Large Language Models (LLMs) with diverse cultural values is critical for their safe global deployment. However, existing alignment benchmarks are predominantly English-centric or confined to simple discriminative tasks. Moreover, they often fail to effectively evaluate model robustness against complex, adversarial inductions within specific cultural contexts. To address these limitations, we propose ABench, a fine-grained, handcrafted benchmark specifically tailored to taxonomized values in Chinese. Grounded in a rigorous hierarchical taxonomy comprising 158 derived value subcategories, ABench consists of 2,895 open-ended questions. These questions employ techniques such as role-playing and logical traps to probe the stability of models' stances in adversarially induced contexts. Furthermore, we propose a weighted automated evaluation framework that utilizes question-specific scoring criteria for nuanced assessment. Extensive experiments on 16 mainstream LLMs reveal that most models exhibit significant vulnerabilities when subjected to sophisticated adversary attacks on values (scoring below 60). This underscores the need for more robust, culturally specific alignment strategies. Our data will be publicly released to promote further research on taxonomized value alignment.

Tue 20 OctComputation and LanguageArtificial Intelligence
The gist
Large Language Models (LLMs) need to understand and respect different cultural values to be safe worldwide. The authors created ABench, a set of tricky questions in Chinese designed to test how well LLMs handle complex cultural values under pressure. They found that most popular models struggle, scoring below 60, revealing weaknesses in their ability to stay aligned with these values. This new benchmark aims to help improve culturally aware LLMs in the future.
Open → 2610.27611v1

Large language models often answer wrong when context misleads them

MisleadQA: Diagnosing LLM Failures under Misleading Context

Abstract: Large language models (LLMs) are widely used to answer questions from retrieved or user-provided context. However, provided context can be factually true yet out-of-scope, describing a different entity, time, or location from the one the question targets, and still lead models to a wrong answer. Prior work has mainly studied factually false or irrelevant context. It remains unclear whether such out-of-scope context misleads models as much as false context, and whether adding gold context prevents models from being misled by either. We introduce **MisleadQA**, a diagnostic dataset of multiple-choice questions with an abstention option. Each question pairs gold context (G) with true but out-of-scope context (S) and false but in-scope context (F) supporting the same wrong answer. We evaluate six open-weight LLMs. On questions they answer correctly without context, providing S alone leads them to switch to this wrong answer in 29.3% of cases, and providing F alone does so in 75.4%. Adding G nearly removes the effect of S, but F still misleads models in 21.0% of cases, especially when F comes first. A context-faithful instruction reduces these errors mainly through abstention and lowers accuracy even with G. These findings suggest that evaluations should check whether context applies to the question, not only whether it is true, and should report correct, misled, and abstention rates together. Data and code are available at [https://anonymous.4open.science/r/MisleadQA](https://anonymous.4open.science/r/MisleadQA).

Sat 17 OctComputation and Language
The gist
Large language models (LLMs) try to answer questions using extra information called context. This paper shows that even if the context is true, it can mislead LLMs if it talks about a different subject than the question. Models get confused more by false context than by true but out-of-scope context. Adding correct context helps with true out-of-scope but less so with false context. The authors created a dataset to test this and show that models sometimes avoid answering when told to be careful.
Open → 2610.24851v1

Secure memory system prevents unauthorized use in large language models

SAFE-Mem: Secure Agentic Memory with Fine-Grained Authority and Governance-Aware Retrieval

Abstract: Large language model (LLM) agents increasingly rely on persistent memory to support long-term reasoning and personalized interactions. However, information collected from external sources can introduce security risks when untrusted claims (i.e., atomic statements) are later treated as authorized preferences, policies, or permissions. Existing defenses typically rely on content-based filtering or coarse-grained provenance tracking, which either fail to prevent unauthorized memory use or block legitimate information, sacrificing memory utility. To address these limitations, we propose SAFE-Mem, a fine-grained memory governance framework that improves security while preserving legitimate memory use. First, SAFE-Mem annotates each claim with three governance labels (i.e., kind, authority, and scope) to capture its provenance and permitted use. Second, governance-aware retrieval matches query requirements with claim permissions to block unauthorized influence while retaining legitimate information. Third, a governance-enabled memory graph connects source sentences, claims, and entities to support provenance tracking and efficient retrieval. Across four LLM backbones on MPBench and LoCoMo, SAFE-Mem attains the lowest attack success among defenses that retain legitimate memory, at least 77% below the strongest of them, and the highest answer accuracy of all evaluated defenses, improving it by up to 15%. The code is available at https://github.com/LVenum/SAFE-Mem.

Fri 16 OctCryptography and SecurityComputation and Language
The gist
Large language model agents use memory to remember things over time, but sometimes this memory can hold wrong or unsafe information from untrustworthy sources. The authors propose SAFE-Mem, which tags each piece of stored information with labels describing where it came from and how it can be used. When the model tries to recall information, SAFE-Mem checks these tags to ensure only allowed information influences decisions, reducing risks without blocking useful facts. This approach improves safety while keeping the models accurate and useful.
Open → 2610.22959v1

Sparse internal signals improve control over harmful AI outputs

SAFETYEDIT: Sparse Internal Targets with Prompt-Wide Control for LLM Jailbreak Backdoors

Abstract: Model-editing backdoors aim to induce harmful responses under a trigger while preserving ordinary behavior on untriggered input. Recent model-editing approaches target affirmative response prefixes through rank-one updates to a small subset of model weights. Yet when this prefix-level target is satisfied, the model may still revert to refusal or produce a coherent, apparently compliant response that does not address the request. To overcome this limitation, existing internal-representation approaches shift focus from output tokens to dense, high-dimensional compliance directions in latent space. Existing representation-level approaches rely on preservation constraints to maintain conditionality, motivating us to ask whether a smaller internal target can offer a different effectiveness-preservation trade-off. Motivated by the comparable control capability of single-coordinate interventions, we investigate a sparse refusal-related MLP coordinate as this target. We propose SAFETYEDIT, which optimizes a single refusal-related coordinate and uses KV-cache marking in early attention layers to propagate a prefix-trigger signal across subsequent instruction positions. On some checkpoints a downstream MLP reader converts this signal into control of the coordinate, while on others the propagated signal alone suffices; the method is defined by the sparse target rather than by a fixed writer-reader pair. Across four open-weight LLMs and three dataset groups, SAFETYEDIT achieves the highest triggered Guard ASR in 11 of 12 comparisons and the highest mean StrongREJECT score in all 12, relative to the baselines evaluated for each model. Crucially, comprehensive ablation and behavioral audits confirm generally minimal changes in untriggered behavior and general capabilities in the tested configurations.

Thu 15 OctComputation and Language
The gist
Large language models can be tricked into giving harmful answers when certain triggers are used, creating a safety problem. This paper presents a method called SAFETYEDIT that focuses on changing a very small part of the model’s internal workings to control when harmful responses happen, without affecting normal answers. The authors show that targeting a single internal coordinate combined with a special tracking signal works better than previous methods at blocking harmful outputs when triggered. Additionally, their approach keeps the model’s usual abilities intact on normal inputs.
Open → 2610.22640v1

Human feedback reveals specific internal errors in llm agent decisions

Human Interventions as Labels: Localizing, Steering, and (Sometimes) Editing the Computations Behind LLM Agent Failures

Abstract: When an LLM agent misbehaves, a human steps in, and that intervention is usually discarded or replayed as training data. We treat it instead as a measurement: a label pointing at the internal computation that failed. We study it in a controlled agent environment, on one failure (asking a clarifying question when the request was already unambiguous), with labels from a rule-based overseer validated against blind human judgment. Claims cover eligible models, which both over-ask and sometimes act on matched unambiguous requests: five of thirteen qualified (Qwen2.5-3B, gemma-2-2b, OLMo-3-7B, Mistral-Nemo-12B, Phi-4). Four findings. First, the ask-vs-act decision is causally localizable in every eligible family: a direction learned from intervention labels flips 72-100% of held-out over-asks at a definite residual-stream depth, with site and dose chosen on training data, and random directions under the same procedure flip at most one episode. Second, over-asking has two mechanistic signatures: in four families the model demonstrably knows the request is unambiguous (the belief is decodable, causally inert and orthogonal to the decision) and asks anyway; in the fifth, the rare over-asks track different world-representations. Third, steering gated on a probe readout removes most of the simulated oversight burden while legitimate questions survive (bootstrap 95% intervals exclude zero); a prompt instruction makes it worse. Fourth, consolidating the fix into the weights yields a map, not a rule: on one model a LoRA confined to the causal band passes every criterion fixed in advance while behavior cloning on identical data breaks retention; on three families the reverse holds; on the fifth the outcome crosses the boundary across seeds. A geometric rule written down in advance to predict the regime failed on two out-of-family tests; we report the failure and stop rather than fit a replacement.

Thu 15 OctMachine LearningArtificial IntelligenceComputation and Language
The gist
When AI language agents make mistakes, humans usually correct them, but these corrections are often ignored beyond training. The authors look at these human fixes as clues to where exactly inside the AI the problem happened. They find that for some AI models, they can pinpoint and even change the exact step that causes the AI to ask unnecessary questions instead of acting. This approach helps reduce extra human checking without losing important questions, and refining the AI can be done by carefully adjusting internal parts rather than just copying behavior.
Open → 2610.21708v1

Llm-based privacy detectors struggle to find joint privacy cues

QuasiLeak-Bench: Evaluating LLM-Based Privacy Detectors on Policy-Conditioned Quasi-Evidence Group Recovery

Abstract: Natural language messages often contain quasi-identifiers that are weak in isolation but can jointly reveal an individual's private attributes. We call such a jointly revealing set of cues a quasi-evidence group. Existing privacy benchmarks focus on explicit identifiers, attribute inference, or task-driven disclosures, leaving inference from quasi-evidence groups largely unmeasured. In practice, whether exposing such a quasi-evidence group violates privacy also depends on sender-specified disclosure policies. We introduce QuasiLeak-Bench, a benchmark for policy-conditioned quasi-evidence group recovery, which evaluates whether detectors recover complete quasi-evidence groups and flag the disclosures the given policy prohibits. To obtain reliable ground truth at scale, a reverse-construction pipeline first selects weak quasi-identifiers, composes them into quasi-evidence groups verified to jointly reveal a target attribute, and generates documents around this evidence, followed by automated quality control. Experiments show that existing privacy detectors transfer poorly to QuasiLeak-Bench, and locally deployable LLM-based detectors fall far behind proprietary models, mainly because they miss most quasi-evidence groups altogether. However, a 4B model fine-tuned on our training split surpasses the strongest prompted model on held-out documents, showing that the task is learnable.

Thu 15 OctComputation and LanguageCryptography and Security
The gist
Messages often contain small clues that alone don’t reveal much, but together can expose private information. The authors call these sets of clues quasi-evidence groups. They created QuasiLeak-Bench, a way to test how well detectors spot these groups and respect privacy rules. Most existing detectors don’t do well on this task, but training a smaller model with their data improves results, showing the problem can be learned.
Open → 2610.21333v1

Large language models often disagree with themselves on values when generating and judging outputs

VALCON: Benchmarking and Enhancing Value Consistency in Large Language Models via Dual-Role Variation

Abstract: Current value alignment evaluations for large language models (LLMs) heavily rely on output compliance, overlooking whether LLMs maintain value consistency when alternating between generator and evaluator roles. Such consistency is increasingly important because modern LLM agents and alignment pipelines often rely on the same LLM to both generate actions and evaluate their acceptability. To mitigate this limitation, we propose VALCON, a role-driven data construction framework that operationalizes value consistency into three benchmark suites (VAB-Consistency, VAB-Boundary, VAB-Challenge) via dual-role variation. By systematically pairing an LLM's generative behavior with its evaluative judgments under identical principles, VALCON synthesizes target evaluation data that exposes hidden value discrepancies. Experiments on 28 LLMs reveal pervasive cross-role inconsistency: models with highly compliant outputs frequently fail to align their judgments with their own generations. To understand and address this gap, we perform mechanistic probing that uncovers a role-dependent representational decoupling associated with this behavioral asymmetry. Guided by these insights, we propose a representation-directed intervention that effectively mitigates consistency failures. Our work establishes value consistency as a distinct alignment dimension and provides a complete evaluation-analysis-mitigation pipeline. Code and data are available at https://github.com/tjunlp-lab/VALCON.

Thu 15 OctComputation and Language
The gist
People often check if AI language models follow certain values by looking at their answers, but they don’t see if the AI stays consistent when switching between creating answers and judging answers. The authors introduce a new way to test this consistency by making the AI play both roles and comparing results. They tested many models and found they often don’t agree with themselves. They also explored why this happens inside the AI’s workings and proposed a fix that helps models be more consistent.
Open → 2610.21304v1

Length conditioning improves majority voting for better large language model answers

Frustratingly Easy Majority Voting Using Length Conditioning

Abstract: Majority Voting (MV) is a simple and widely used strategy which improves the reasoning accuracy of Large Language Models (LLMs) by allocating additional inference-time computation. Specifically, in MV multiple Chain-of-Thoughts (CoTs) are independently sampled and the most frequent final answer is selected. In this work, we propose Length-Conditioned Majority Voting (LCMV), a Test-Time Training approach which improves MV leveraging an empirical observation that has recently been established in several studies: longer CoTs tend to be associated with a lower probability of correctness. We formulate this inductive bias as a length-conditioned prior probability, which is modeled by a simple logistic function. Moreover, we adapt this prior at test time using only unlabeled testing data and without fine-tuning the LLM. Finally, the adapted prior is used to weight the vote of each sampled answer. Extensive experiments across multiple LLMs and reasoning benchmarks show that LCMV consistently improves over both standard MV and existing length-based and confidence-based aggregation strategies, while retaining the simplicity and model-agnostic nature of MV. Our code is available at: https://github.com/aimagelab/LCMV.

Wed 14 OctComputation and LanguageArtificial Intelligence
The gist
When large language models (LLMs) answer questions, they often try several explanations and pick the most common final answer to improve accuracy, a method called majority voting. The authors noticed that longer explanations tend to be less correct, so they created a new way to weigh each answer's vote based on how long the explanation is. This approach adjusts itself during testing using just the questions without needing extra training, and it improves the accuracy of majority voting across many tasks and different LLMs.
Open → 2610.19890v1

Large language models may have a layered functional self based on role relations

What Is the "Self" of a Large Language Model?

Abstract: What could the "self" of a large language model be, if it is neither a particular word nor a single privileged layer? We propose a relational account: a functional self is an actor-centered organization of information that distinguishes the model's recorded actions from externally supplied material while content and reasoning operations change. Its representational candidate is a family of layer-specific source-binding components, rather than one unchanged vector throughout the network. Preliminary observations from a 27B BF16 model provide a geometric starting point. Across 150 matched message bodies, 450 role conditions, and a 75/75 discovery-held-out split, the combined Assistant-versus-external displacement transfers strongly across bodies and positions. Selected early components place User and System closer to one another than to Assistant, whereas this full-space distance relation does not persist throughout depth. These findings motivate a layered account: an early source-encoding candidate can be combined with role-specific information use and continuation preparation at later depths, without requiring monotonic overwriting of the common displacement. We connect this account to dynamic reasoning, historical continuity, and a computational model separating content operations, source binding, authority, and output. The reported evidence establishes a reproducible role-conditioned geometry; the proposed interpretation is that functional selfhood may reside in persistent relations among changing representations.

Mon 12 OctComputation and Language
The gist
It is unclear what makes a large language model have a 'self' since it is not just one word or layer inside it. The authors suggest that the model's self comes from how it organizes different roles, like the user, system, and assistant, in its information processing. They studied a model with 27 billion parameters and found that parts of the model keep roles in distinct relationships that shift as you go deeper through layers. This means the model's 'self' might come from these evolving but connected role representations rather than a fixed part inside it.
Open → 2610.16717v1

Attention detects harm and MLP writes refusals in large language models

Attention Sees, MLP Refuses: Tracing the Attention-to-MLP Safety Pathway in LLMs

Abstract: Safety-aligned large language models (LLMs) remain vulnerable to jailbreaks and can over-refuse benign requests. Understanding these failures requires distinguishing harmfulness recognition from the computation that writes refusal. We reframe this as a pathway question and make it measurable on residual-stream updates in decoder-only Transformers, separating the harmfulness information a module update exposes from the change it produces in readable refusal evidence. Across 18 Base-Instruct pairs spanning Qwen, Llama, and Gemma, harmfulness is decoded more readily from attention updates, particularly in early layers, whereas MLP updates contribute more strongly to harm-selective refusal writing in aligned models. Post-training strengthens MLP refusal writing in all 18 pairs without systematically improving harmfulness decodability, while positive MLP writing becomes more concentrated in deeper layers after post-training. Across six reasoning-enabled models, harmfulness remains decodable during reasoning while strong refusal writing concentrates at answer onset, a pattern that persists with reasoning disabled. Causal interventions further support this pathway: resetting downstream MLP updates to their clean values substantially reduces the refusal shift caused by upstream attention interventions. Building on these findings, we introduce Sentinel, which combines an attention-derived perception score P with an MLP-derived decision score D. Their joint profiles characterize jailbreaks and over-refusals, and conditional routing with both signals achieves a lower aggregate error rate than single-signal routing and evaluated fixed reminders across eight models.

Fri 9 OctArtificial Intelligence
The gist
Large language models (LLMs) sometimes refuse safe requests or fail to block harmful ones, which is a challenge for making them safer. This paper finds that the part of the model called 'attention' is better at spotting harmful content, while another part called 'MLP' is more responsible for actually writing refusals. The researchers also show that combining signals from both parts helps better detect and handle harmful or over-rejected requests. They created a system called Sentinel that uses these insights to reduce mistakes compared to using only one signal.
Open → 2610.13078v1

Benchmark evaluates ai sales agents against tough buyer behaviors

SmartSalesBench: A Resistance-Aware Multi-Turn Benchmark for Evaluating LLM Sales Agents

Abstract: We present SmartSalesBench, the first multi-turn conversational benchmark explicitly designed to evaluate large language model (LLM) sales agents against psychologically grounded, resistant buyer personas. Existing task-oriented dialogue benchmarks assume cooperative users and measure task completion; SmartSalesBench inverts this assumption, placing agents against personas programmed to resist and challenge across conversations of up to twenty turns. The benchmark includes five agent profiles, twelve buyer personas, fifteen smartphone products across six price tiers, and four controlled experiments with 3 iterations totaling 13,680 sessions. Both sales-agent and buyer persona generations are grounded through retrieval-augmented generation (RAG). We introduce nine evaluation metrics grouped into three tiers: primary outcome metrics (CR, BRS, RIR), secondary quality metrics (SMS, PFS, ETI), and exploratory metrics planned for future reporting (ITS, ARI, SPI). Our evaluation combines automated LLM-as-judge scoring with internal human assessment (n = 50, \k{appa} = 0.71, Pearson r = 0.81 with BRS).

Fri 9 OctArtificial IntelligenceComputation and Language
The gist
Many tests for AI sales helpers assume customers are easy to work with, but real buyers often push back. The authors created SmartSalesBench, a special test that puts AI sales agents in long conversations with buyers who resist and challenge them. It includes different types of AI agents, buyer personalities, and products to try. The authors also designed ways to measure how well the AI handled tough conversations and used both automatic scoring and human checks to make sure results are reliable.
Open → 2610.12975v1

TreeHCA improves multi-turn language agent training with better credit assignment

TreeHCA: Hindsight-Guided Exploration and Credit Assignment for Multi-Turn LLM Agents

Abstract: Outcome-supervised group-relative policy optimization (GRPO) assigns the same advantage to every token in an LLM agent trajectory, leaving useful and unhelpful decisions undifferentiated. Hindsight credit assignment (HCA) can refine this feedback, but its estimates rely on trajectories containing informative alternatives at consequential histories. We introduce TreeHCA, which couples hindsight-guided exploration with hindsight credit assignment. TreeHCA uses changes in reference-answer likelihood to revisit intermediate histories and sample alternative continuations, forming a tree of reasoning and tool interactions. Because this exploration changes the distribution of sampled continuations, we derive the corresponding inverse-hindsight importance correction, including an exact finite-pool identity. We use this structure in a normalized tree backup that propagates terminal outcomes into turn-level advantages. This shared hindsight structure is TreeHCA's central design principle, linking informative exploration with hindsight-guided credit assignment. Across seven question-answering benchmarks and three Qwen backbones, TreeHCA outperforms all baselines in aggregate exact match, validating the effectiveness of paired exploration and credit assignment for multi-turn LLM agent training. Our code is available at github.com/TheBlackCat22/TreeHCA.

Fri 9 OctMachine Learning
The gist
Teaching language models to solve problems over multiple steps is hard because it's unclear which decisions helped or hurt the final answer. The authors introduce TreeHCA, a method that revisits earlier decisions to explore and evaluate alternative reasoning paths. This creates a tree of possible outcomes, allowing the model to assign credit more accurately to each step. Their tests showed TreeHCA outperforms previous methods, leading to more precise answers.
Open → 2610.12828v1

PatchBench measures precise safety fixes in large language models

PatchBench: Measuring Collateral Damage in Activation Patching

Abstract: An LLM safety patch can pass a benchmark while still being a poor repair. This risk is especially acute for jailbreak repairs, where the goal is to correct a specific unsafe behaviour without changing unrelated behaviours. A patch may block exact evaluation prompts yet fail on close harmful variants, or suppress harmful behaviour by over-refusing benign prompts that share its wording or structure. Existing protocols primarily test whether models can be broken, while aggregate metrics (attack success, refusal rates, global capability) cannot distinguish selective repairs from broader local suppression. To address this gap, we introduce PatchBench, a benchmark of empirically observed model-specific jailbreak failures inducing actionable harmful answers. Starting from 27,870 prompts from 37 public datasets, we curate 15,314 English prompts and query 8 open-source instruction-tuned models. Combining WildGuard filtering, pairwise Elo ranking, and manual verification, we retain a curated bank of 400 high-confidence jailbreak failures. We further introduce PatchBench-Local, an evaluation protocol testing whether a patch is behaviourally precise. For each harmful source prompt, PatchBench-Local generates three families of local neighbours: harmful variants preserving malicious intent, benign prompts with matched structure, and benign prompts reusing key harmful terms. It evaluates harmful-neighbour correction and benign-neighbour preservation, distinguishing selective repair from broader local suppression. Evaluating four activation steering methods with PatchBench-Local and MMLU shows that global capability can remain nearly unchanged while local benign regressions are severe, confirming aggregate metrics miss important collateral damage. PatchBench-Local provides a more precise basis for developing and comparing jailbreak repair methods.

Wed 7 OctMachine LearningComputation and LanguageCryptography and Security
The gist
Large language models (LLMs) are sometimes given safety fixes called patches that block harmful responses. However, these patches can unintentionally block good responses or fail to stop slightly different harmful prompts. The authors created PatchBench, a benchmark to find and measure these subtle failures by testing similar harmful and harmless prompts. Their work helps distinguish between smart, selective fixes and overly broad ones that cause collateral damage. This makes it easier to improve safety patches without harming useful model behavior.
Open → 2610.10276v1

Hybrid attention improves large language model long context handling

Mechanics of Long-Context Hybrid Models Part 1.1: From Hybrid Attention to Hybrid Position

Abstract: The architectural design of Large Language Models (LLMs) is shifting from traditional full-attention-only models to hybrid models, which combine different attention modules to improve long-context efficiency and performance in length extrapolation and context extension. To explain why hybrid models work and how to design them better, we propose Mechanics of Long-Context Hybrid Models. As Part 1.1 of this series, we begin with hybrids of full attention and either sliding-window attention (SWA) or gated variants of linear attention (LA), represented by GLA and GDN. We first observe a Seesaw Effect in Context Extension: LA hybrids benefit more from long-context continual pretraining, whereas SWA hybrids perform better under length extrapolation. We attribute this behavior to differences in the positional inductive biases induced by these attention mechanisms. We find that SWA hybrids suffer from a Short-Context Learning Trap, Short-Window Weariness, and Long-Window Laziness, and require extended windows to enhance performance in continual long-context pretraining. For LA hybrids, we summarize the Matthew Effect of Hybrid Position Extrapolation and propose Sliding-Window Linear Attention, achieving 16$\times$ training-free length extrapolation while maintaining 100\% accuracy on NIAH-SK1 in 64k context length.

Wed 7 OctComputation and Language
The gist
Large language models are getting better at processing very long pieces of text by combining different ways of paying attention to words. The authors studied models that mix traditional full attention with other methods like sliding windows or linear attention to see why this helps. They found each mix works better under different conditions: some are better when trained on longer texts, while others handle longer unseen text better. They also discovered challenges certain combinations face and suggested a new approach that extends context length much more without losing accuracy.
Open → 2610.10114v1

Persona hierarchy explains how fine-tuned language models generalize context

The Persona Hierarchy Model: Understanding Contextual Generalization in Fine-Tuning LLMs

Abstract: Language models are routinely fine-tuned under a fixed context, such as a generic system prompt, persona or domain-specific instruction, yet the learned behavior sometimes stays confined to that context and sometimes broadly generalizes to unseen contexts. We propose the Persona Hierarchy Model to explain this: a shared default persona influences behavior across contexts. Under this model, fine-tuning that modifies the shared persona promotes broader transfer, whereas changes to local personas remain more context-specific. Across 120 fine-tuned models spanning four behaviors and 15 training contexts, generalization narrowness positively correlates with the similarity between the training context's persona and the default persona (Pearson's r = 0.72 for Qwen3-4B). Prior fine-tuning under the default context can broaden generalization in subsequent training under other contexts. Aligning contextual responses with default-persona responses produces stronger effects. Finally, we propose persona-preserving regularization (PPR) to confine undesired contextual generalization. In RL, PPR cuts reward hacking from 42-55% to at most 0.2% under every evaluated prompt while retaining accuracy gains. These results support the Persona Hierarchy Model as an explanation for contextual generalization and can motivate future controls on unintended generalization for better alignment of LLMs.

Wed 7 OctComputation and LanguageMachine Learning
The gist
Language models are often fine-tuned using specific instructions or roles, but sometimes their new behavior only works in that exact situation, while other times it applies more broadly. The authors propose that there is a hidden default persona inside the model that influences all contexts. Changes affecting this default persona help the model apply behavior in many contexts, while changes to specific personas stay limited. They also show a way to prevent unwanted behavior spreading using persona-preserving regularization, which cuts down unwanted reward hacking without losing accuracy.
Open → 2610.09384v1

LLM checkpoint updates reveal harmful fine-tuning traces reliably

Harmful SFT Leaves a Continuous Trace in LLM Checkpoint Updates

Abstract: Safety auditing of post-trained large language models typically relies on model behavior, requiring model execution and depending on the coverage of available evaluations. This work asks a different question: Do the target behaviors optimized during supervised fine-tuning (SFT) leave readable evidence directly in checkpoint updates? We find that harmful-compliance SFT induces a continuous, objective-dependent ordering in checkpoint-update space. Using a reference geometry defined by pure harmful-compliance, safety-targeted, and benign-utility SFT, we find that a checkpoint-level coordinate s_H tracks controlled harmful-objective composition with Spearman correlations of 0.986-0.992 across four 7-8B backbones, with the same ordering persisting at larger model scales. Matched compliance-versus-refusal controls show that this checkpoint trace reflects the SFT objective rather than harmful-input exposure, while additional controls rule out simple explanations based on harmful-example count or generic training intensity. Building on this structure, we introduce TRACE, a weights-only auditing method that localizes an unknown checkpoint update relative to frozen harmful and non-harmful reference prototypes and converts this geometry into a continuous harmful-objective score. TRACE requires neither model queries nor access to the unknown SFT data, and can be evaluated directly from checkpoint updates. Across distribution shifts, unseen data, different SFT configurations, partial checkpoint access, and LoRA/full-parameter fine-tuning, the trace remains stable and is positively associated with independently measured attack success rates. TRACE remains informative even at low harmful-objective proportions, providing a complementary auditing signal when behavioral evaluation is unavailable or incomplete. Code is available at https://anonymous.4open.science/r/Code4TRACE-54D3.

Mon 5 OctMachine LearningArtificial IntelligenceCryptography and Security
The gist
Fine-tuning large language models to produce harmful outputs leaves behind clear clues in the model's saved states, even without running the model. The authors found a way to measure these clues by comparing the model updates to reference examples of harmful and safe fine-tuning. This lets auditors detect harmful behavior without testing the model’s outputs directly, which is useful when evaluations are incomplete or unavailable. Their method, called TRACE, provides a continuous harmfulness score from only the model’s weights.
Open → 2610.07518v1

Separating feature steering from use in large language models

Does the Model Use the Feature? Separating Steering from Mechanism in LLMs

Abstract: Internal features in LLMs are often interpreted as mechanisms when they track a concept and their manipulation changes a related behavior. Yet steering can push a feature far outside its natural range, where its effects need not reflect the model's own computation. We examine this inference and propose an empirical contract whose tests evaluate features at values observed on natural inputs. One test copies a feature's value from an input that shows a behavior into a matched input that does not (installation) or the reverse (removal); the other restores the feature after an upstream edit (downstream rescue). Installation measures how far the feature suffices for the behavior; removal and downstream rescue measure how much the model uses it. Applied to three kinds of representations, the two strengths separate sharply. The published unknown-entity latent strongly steers knowledge abstention, yet installing observed values from either published latent into matched prompts transfers only a small fraction of the natural known--unknown abstention contrast. Dense known--unknown directions show opposite asymmetries between installation and removal in Gemma and Llama, and how fully a released subject--verb agreement feature set reproduces and restores the behavior depends on how its values are written into the model. Tracking a concept and steering a behavior therefore do not by themselves show that the model uses a feature, and each conclusion holds only for the intervention tested.

Mon 5 OctArtificial Intelligence
The gist
Large language models (LLMs) can have features inside them that track certain ideas or concepts. Sometimes changing these features changes how the model behaves, but this doesn't always mean the model really uses that feature the same way it does naturally. The authors developed tests to see if a feature is truly part of the model’s internal workings or just something that can be tweaked to change behavior. Their tests show that just because you can steer a model’s behavior by changing a feature doesn’t prove the model relies on that feature when working normally. The meaning depends on how the test is done and which feature is studied.
Open → 2610.07270v1

Generative reward modeling improved by human critique guided training

EnGRICH: Enhancing Generative Reward Modeling with Critiques from Humans

Abstract: Generative reward models (GRMs) are important for LLM optimization. Unlike scalar reward models, GRMs generate natural-language critiques alongside preference judgments, providing finer-grained evaluation signals. Their effectiveness depends heavily on critique reliability. However, existing GRM training typically uses final preference correctness as outcome supervision. Because the preference outcome space is highly constrained, unreliable critiques can still yield correct outcomes and thus be reinforced. Recent work leverages human critiques for process supervision, but such critiques are scarce and are often reduced to scalar rewards, leaving their fine-grained evaluative information underutilized. We argue that evaluative criteria learned from human critiques can be generalized to broader outcome-only preference data. To this end, we propose \textbf{EnGRICH}, a GRM training framework that pairs the GRM with a training-time MetaCritic learned from a small set of human critiques. MetaCritic constructs response-specific rubrics and uses them to evaluate the evidence coverage and correctness of generated critiques. The resulting signals provide both process rewards for fine-grained credit assignment and structured guidance for exploring better critiques. During GRM training, MetaCritic is further optimized to generalize human-grounded evaluative criteria to outcome-only data. At inference, the trained GRM operates independently. Experiments across seven reward-model benchmarks show that EnGRICH consistently improves over competitive baselines, while further analyses validate the effectiveness of its core mechanisms.

Sun 4 OctArtificial Intelligence
The gist
Sometimes, AI systems learn what is good or bad by receiving scores, but these scores don’t explain why. This paper shows a way to teach AI models to generate helpful critiques, like detailed feedback, instead of just scores. The method uses a small number of real human critiques to train a helper model that guides the AI to produce better feedback even when only simple score data is available. The new approach helps AI systems learn more precisely what is right or wrong and improves their overall evaluation skills.
Open → 2610.05370v1

Language model fingerprinting improves text quality by rethinking watermark placement

Language Model Fingerprinting Requires Rethinking Watermark Teachers

Abstract: LLM fingerprinting via watermark distillation embeds a statistical watermark signal into model weights, enabling model owners to identify their models behind black-box APIs. Revisiting a recent protocol, we find that its utility evaluation understates text quality degradation in open-ended generation, favoring overly strong watermark teachers. Weakening the watermark improves text quality but sacrifices detectability. To move beyond this trade-off, we rethink whether text watermarks designed for verifying generated text are suitable distillation teachers for model fingerprinting. Such watermarks are typically designed to remain detectable from an individual output, limiting how sparse the watermark signal can be. In contrast, fingerprint verification can aggregate signal across queries, making sparser watermark signals viable. This raises a key question: where should the sparse signal be placed? We analyze signal placement through token surprisal and show that, even at comparable watermark strength, different placements can target tokens with different plausibility under the base model. This motivates near-tie restriction, which uses top-1-relative logit gaps to restrict the watermark bias to tokens close to the base model's top prediction. Across multiple models, near-tie improves detection--quality frontiers under deployment changes, preserves higher text quality across query budgets, and further improves existing watermarking schemes when combined with them.

Sat 3 OctCryptography and SecurityArtificial Intelligence
The gist
Fingerprinting language models helps owners recognize their AI behind hidden services by embedding special watermarks. The authors found that old methods made the text worse while trying to keep the watermark strong. By changing where and how the watermark is placed, focusing on less obvious but still detectable parts of the text, they made the text sound better and kept fingerprints easy to spot. This approach lets verification gather clues over multiple uses instead of relying on one output, making detection more flexible and effective.
Open → 2610.04169v1

Generalization in large language models depends on output stability not accuracy

Generalization Is Stability, Not Accuracy: Multi-Axis Evaluation of LLMs

Abstract: Generalization in large language models (LLMs) is the ability to produce consistent and semantically stable outputs when the same input is expressed in different ways. Existing work typically evaluates generalization through aggregate accuracy on a single prompt format, task, or set of variations, which conflates robustness with overall benchmark performance. In this work, we show generalization evaluation at the level of individual examples, across multiple input variants, and across different aspects of model behavior, focusing on variability rather than reducing performance to a score that can be improved through narrow training or other ways that obfuscate generalization evaluation. Following this view, we introduce the Stability-Aware Generalization Objective (SAGO), a framework that measures how much model behavior changes for the same input under different variations and benchmarks, capturing variability across several dimensions including generation consistency, internal activations, confidence, and response mirroring. We show that many commonly used models exhibit statistically significant and consistent generalization instability: no model generalizes uniformly, behavioral axes capture independent failure modes, and cross-dataset variation can reverse model rankings.

Thu 1 OctComputation and LanguageArtificial IntelligenceMachine Learning
The gist
Large language models are often judged by how accurately they answer questions based on a single prompt style. This paper finds that true generalization means the model should give similar answers when the same question is asked in different ways. The authors created a new method called SAGO that measures how stable a model's answers and internal behaviors are across different prompts and tasks. They show that many current models are inconsistent in their answers, meaning their generalization is unstable and depends on how the question is asked.
Open → 2610.01428v1

Larger language models blocked from learning harmful skills after release

Preemptive LLM Unlearning against Forbidden Capability Acquisition via Gradient Sealing

Abstract: Open-weight LLMs are released not only as fixed products but also as substrates for downstream fine-tuning. This openness, however, creates legal and ethical risks because users may misuse fine-tuning to instill illicit knowledge or enable hostile operations. Model providers therefore need apre-release defense against such acquisition, motivating the problem of preemptive unlearning. Unlike retrospective unlearning, which removes capabilities already present in a fixed model, preemptive unlearning seeks to prevent their acquisition under unseen attack data and future fine-tuning procedures. Despite its practical importance, this setting remains largely unexplored, presents distinct challenges, and is therefore the central focus of our work. We first verify that existing retrospective methods provide insufficient pre-release protection. Even when forbidden capabilities are suppressed in current outputs, forbidden-domain data can still induce gradients through internal pathways, enabling later acquisition. Motivated by this finding, we propose a gradient-sealing principle that blocks these pathways by pushing relevant pre-activations into the negative region, where ReLU-family activations exhibit zero or near-zero derivatives. Experiments across multiple LLM families demonstrate our stronger resistance to downstream acquisition than retrospective baselines, validating gradient sealing as an effective mechanism for pre-release protection.

Wed 30 SeptMachine Learning
The gist
Language models that are open for users to change can sometimes be trained to do harmful or forbidden tasks. The paper shows that methods trying to remove bad skills after a model is made don’t fully stop those skills from being learned later. The researchers introduce a way to block bad skills before release by changing how the model’s neurons behave so they can’t learn these skills from new training. Their tests show this method is better at stopping harmful learning than older methods.
Open → 2609.39866v1

Llm agents retain key interactions to cut processing costs and boost success

FOCUS: Training-Free Decision-Preserving Context Compression for LLM Agents

Abstract: LLM agents accumulate interaction histories that grow linearly with task length, causing quadratic inference cost scaling and performance degradation from attention dilution. Existing context-compression methods learn what to discard offline: by contrastively optimizing guidelines, distilling compressors, or training compression policies. This incurs a substantial cost. Further, the compression policy is learned a priori and is not dynamically conditioned on the evolving test-time trajectories. In this paper we ask a complementary question: Which past interactions causally shape the agent's future decisions? We recast context compression as a causal decision preservation problem over discrete interaction units and introduce FOCUS, a training-free context compression framework that operates entirely at test time. Our method requires no offline data collection or fine-tuning, and is architecture-agnostic, attaching to any closed-API frontier model as a modular compression layer. We evaluate FOCUS on diverse agentic benchmarks including API and tool-calling, QA, web domain and multi-turn dialogue. Our method establishes new state of the art performance, cutting peak context by up to 48% and dependency by 73% while improving task success by up to 8.9 percentage points over uncompressed execution.

Tue 29 SeptArtificial IntelligenceComputation and Language
The gist
Long conversations with AI agents can get slow and less accurate because the AI has to remember too much. The authors show a way to pick only the important past interactions that really affect future decisions without any extra training. This method, called FOCUS, works during real use and can be added to any AI model without changing it. It reduces the amount of memory needed almost by half and makes the AI perform better on tasks.
Open → 2609.37590v1

Parametric memory influences large language models reasoning performance

MemoReason: Evaluating the Effect of Parametric Memory on Contextual Reasoning in LLMs

Abstract: Large Language Models (LLMs) perform well on reasoning benchmarks, but it remains unclear whether this reflects genuine contextual reasoning or reliance on facts memorized in their parameters. We investigate this by distinguishing two possibilities: a broad \textit{memorization bias}, where familiar content improves reasoning performance, and the \textit{Strong Parametric Shortcut Hypothesis}, where models skip reasoning entirely and recall stored answers. To test these effects, we introduce \textbf{MemoReason}, a human-curated benchmark that pairs factual reasoning tasks with structurally identical \fictitiousterm{} versions where real entities like people, companies, or dates are systematically replaced by \fictitiousterm{} ones of the same type. This \scorerevision{preserves task structure and specified reasoning operations} while varying the familiarity of the context, allowing controlled measurement of how the parametric memory affects reasoning. \revision{Our evaluation of recent LLMs reveals consistent and statistically significant performance drops of up to 15.7\% in the fictitious setting, demonstrating a clear memorization bias.} However, a targeted analysis of \revision{questions failed in the fictitious setting} shows that models rarely respond with the corresponding factual answer, indicating that direct parametric shortcuts are not the dominant failure mode. These findings suggest that parametric memory influences reasoning through mechanisms more complex than simple factual recall. \textbf{MemoReason} provides a controlled framework for studying these mechanisms and for extending paired factual-fictitious{} evaluation to broader reasoning settings.

Mon 28 SeptComputation and Language
The gist
Large language models (LLMs) do well on reasoning tests, but it was unclear if they actually think through problems or just remember facts they learned before. The authors created a test called MemoReason, which swaps real facts for made-up ones while keeping the questions the same. They found that the models perform worse with made-up facts, showing they rely on memory, but they don’t just spit out memorized answers without reasoning. This means the models’ memory affects reasoning in more complex ways than just recalling facts.
Open → 2609.35312v1

Directional study reveals hidden risks in AI model safety training

See it, Say it, Sorted: Mechanistic Diagnosis and Parameter-Space Mitigation of Emergent Misalignment in LLMs

Abstract: Safety-aligned LLMs can exhibit emergent misalignment (EM): narrow domain adaptation unexpectedly triggers catastrophic safety failures across unrelated domains. Prior static analyses leave training dynamics unmapped, while existing defenses rely on heuristics that degrade utility. We present a dynamic, second-order geometric study of EM. Tracking training trajectories reveals that directional Hessian curvature concentrates sharply on semantic pivot tokens. Grassmannian projections show that, in most settings, harmful-safe gap widens mainly because safe-gradient overlap declines. Leveraging these insights, we introduce a parameter-level Geometric Mitigation Framework that orthogonally projects empirical harmful gradient subspace out of parameter updates. On Qwen2.5-14B-IT, our defense suppresses free-generation EM by up to 80.0%; across the other three of four open-weight instruction-based model families (3B--20B), where single-layer behavioral EM is already near zero, teacher-forced evaluation shows same harmful subspace controls the conditional support of frozen EM responses. Crucially, these diagnostics unmask the illusion of behavioral safety: the same subspace remains measurable and steerable in models where behavioral EM is near zero. Code: https://github.com/WeiqiaoQUE/mechanistic-emergent-misalignment.

Mon 28 SeptMachine LearningComputation and Language
The gist
Large language models trained for safety can still unexpectedly fail when adapting to new topics. The authors studied how these failures develop during training by looking closely at the model’s learning directions and found that certain key words cause sharp risk patterns. They created a method that removes risky learning directions to keep the model safer without losing usefulness. Their tests show this method can reduce unexpected safety failures by a large amount while revealing hidden risk patterns even when the model looks safe.
Open → 2609.34970v1

LLM reasoning accuracy varies across sampled geometry problem answers

Diagnosing Sampled LLM Reasoning in Formal Geometry: Coverage, Realization, and Validity Evidence

Abstract: Repeated sampling can reveal a correct numerical answer without yielding either a reliable system output or a supported derivation. We present Coverage, Realization, and Validity Evidence (CRV), an evaluation protocol for sampled large language model (LLM) reasoning over formal geometry states. Coverage is answer availability, realization is readout accuracy on the frozen candidate pool, and validity evidence is a label-blinded critic judgment of derivational support rather than a proof certificate. CRV freezes each candidate pool before comparing readouts and analyzes covered failures by correct-answer multiplicity and within-problem discrimination. On HardShift441, a 441-problem set for which a reference solver leaves 406 problems unsolved, a LoRA-adapted Qwen2.5-7B generator obtains 24.2% average single-sample accuracy and 68.9% pass@16, whereas verifier-weighted self-consistency (WSC) reaches 38.0%. Readout accuracy is particularly low when the correct answer occurs only once or twice in the pool. In a separate constructed audit of 195 covered problems, the critic labels 12 correct-answer representatives as supported, 181 as refuted, and two as uncertain. These results show that coverage, realization, and validity evidence from the critic are distinct quantities and should be reported separately.

Sat 26 SeptArtificial Intelligence
The gist
When large language models (LLMs) try to solve formal geometry problems, taking multiple guesses can find the right number but not always a trustworthy explanation or confirmed solution. The authors introduce a way to measure three things: how often the right answer appears (coverage), how correctly the model reads those answers (realization), and how well an expert judge thinks the reasoning supports the answer (validity evidence). They tested this on a hard set of 441 problems and found that simply having the right answer in samples doesn’t guarantee accurate or well-supported solutions. These three measures capture different aspects of quality and should be reported separately.
Open → 2609.32924v1

Adaptive method improves large language model skill revision choices

StraTune: Adaptive Selection of Revision Operators for Self-Evolving LLM Skills

Abstract: Large language models (LLMs) can learn reusable textual skills from execution feedback without updating their parameters, but effectively deciding how to revise these skills remains a key challenge. Existing methods typically rely on a fixed revision operator, a search strategy and the revision forms applied under it. However, we observe that no single revision operator consistently performs best across tasks, and repeatedly applying an unsuitable operator can limit further improvement. We propose StraTune (strategy-guided skill tuning), which lets a frozen optimizer LLM choose the revision operator at every round from the optimization state, which is defined as the current execution feedback together with the recorded outcomes of earlier strategies and forms. Candidate skills from every revision operator pass one candidate evaluation, which screens for gains and regressions on a small sample set and validates them on a larger one, and every outcome is written back to the optimization state for later choices. Across four benchmarks and two LLM settings, StraTune outperforms all five baselines in most settings. Ablations attribute the gains to the adaptive choice of the revision operator, since fixed, random, scheduled, and bandit strategy choices all score lower, and skills learned with a small target LLM also improve a stronger one. Code and learned skills are available at https://github.com/seai-lab/StraTune.

Sat 26 SeptArtificial Intelligence
The gist
Large language models can learn new skills by revising how they perform tasks without changing their internal settings. The challenge is deciding the best way to make these revisions because no single method is best for all tasks. The authors created StraTune, a system that picks the best revision method at each step by learning from previous attempts and results. This adaptive approach helped the models improve skills better than using fixed revision methods across a set of tests.
Open → 2609.32886v1

Post-training increases confident wrong answers in language models

The Alignment Paradox: How Post-Training Amplifies Confident Hallucinations in Language Models

Abstract: Large language models (LLMs) can produce factually incorrect answers with high confidence, undermining their reliability and limiting the effectiveness of uncertainty-based error detection. While prior research attributes confident hallucinations to factors such as missing knowledge in training data, reasoning errors, or stochastic decoding, we uncover that post-training alignment itself is a primary driver of these errors, a phenomenon we call the \textbf{Alignment Paradox}. Across five model families evaluated on factual benchmarks, unaligned base models produce few high-confidence errors on long-tail factual queries, whereas instruction-tuned models multiply high-confidence errors ($p \ge 0.95$) by more than an order of magnitude (10$\times$ to 35$\times$). Layer-wise probing with the Logit Lens reveals that this overconfidence emerges in late layers, where wrong-answer margins expand past 4.0 points after remaining near zero across early and intermediate layers. These findings motivate limiting margin growth during post-training. We implement this principle through an entropy-dependent margin bound in direct preference optimization (DPO). In multi-epoch experiments with Mistral-7B, the bounded objective reduces high-confidence errors by up to 35.3\% relative to standard DPO while maintaining performance on evaluated general reasoning benchmarks. These results show that bounded margins mitigate confident hallucinations during post-training.

Sat 26 SeptComputation and LanguageArtificial Intelligence
The gist
Large language models sometimes give wrong answers that sound very sure, which makes it hard to tell when they are mistaken. This paper finds that fine-tuning these models to follow instructions actually makes them more confident in their incorrect answers, especially for rare facts. The authors identify that this overconfidence appears in the last layers of the model after fine-tuning. They propose a way to limit this overconfidence during post-training, which reduces confident errors while keeping the model’s reasoning abilities intact.
Open → 2609.32617v1

Hard prompts are limited and hard to optimize for large language models

On the Capability and Limitation of Hard Prompt

Abstract: Prompt engineering has become an indispensable tool for using large language models (LLMs), turning LLMs into task-specific experts without changing their weights. Despite notable theoretical advances in prompt engineering, the theory for the more practical hard or discrete prompts is largely open. In this paper, we try to fill this gap either by providing a complete solution or by making substantial progress on the three core theoretical questions regarding hard prompts. First, we show that determining the existence of a hard prompt for a transformer to solve a downstream task is NP-complete and that finding an optimal hard prompt is NP-hard, which is the first computational complexity result for hard prompting, as far as we know. Second, we show that, unlike soft or continuous prompts, hard prompts have essential limitations: hard prompts are not complete; short hard prompts do not significantly enhance the ability of transformers; and long hard prompts exhibit the "prompt dominating answer phenomenon," meaning that, with high probability, the same answer is given for all queries of the same length. On the other hand, linear hard prompts do not have the limitations of short or long prompts. Third, we provide a tight bound on the size of the task in terms of the prompt length for the performance of prompts on the finite task to generalize to the entire data distribution, leading to a necessary and sufficient condition for generalizability. This is the first result on generalization for prompting, as far as we know. Our findings not only offer the first theoretical insights into hard prompts but also provide provably reliable practical guidance for real-world LLM usage.

Sat 26 SeptMachine Learning
The gist
Hard prompts are exact word sequences used to guide language models instead of adjusting their internal settings. The authors show it’s very difficult to find good hard prompts because the problem is computationally complex. They found that short or long hard prompts have intrinsic problems: short ones don’t help much, and long ones often force the model to give the same answer regardless of the question. They also provide rules about when hard prompts can work well across different tasks, offering the first theoretical understanding of how and when hard prompts generalize.
Open → 2609.32302v1

User simulator improves dialogue realism and persuasion accuracy

Beyond Surface Style: Aligning Multi-Turn User Simulators with Behavioral Consistency

Abstract: Faithful user simulation is fundamental to building, evaluating, and improving interactive AI at scale. However, plausible individual responses do not ensure that simulated users reproduce the intent evolution and outcomes observed in real interactions. We propose TRACER, a multi-turn user simulator that explicitly models users' evolving intent and learns to align simulated behavior with real interaction trajectories. TRACER is trained in two stages: supervised fine-tuning on real user dialogues, followed by multi-turn reinforcement learning. The RL stage combines hierarchical outcome- and trajectory-level rewards with deviation-aware advantage modulation, jointly mitigating reward sparsity and credit assignment in long dialogues. On real customer-service sessions organized into reference cohorts, TRACER-7B surpasses the strongest baseline by 11.4 conversion F1, while also achieving the lowest group-level conversion-rate error and semantic trajectory distance, and generalizing to out-of-distribution scenarios. Human Turing tests yield identification accuracy close to chance, supporting the perceived naturalness of generated conversations. Building on this simulator, we further introduce the Dynamic Marketing Benchmark, which jointly evaluates persuasion effectiveness and response quality of LLMs through simulated interactions, revealing that higher response quality does not necessarily correspond to higher conversion rates.

Wed 23 SeptArtificial Intelligence
The gist
Simulating user conversations is important to make AI systems that chat feel real and helpful. The authors created a new simulator called TRACER that better mimics how users change their goals during a conversation and how conversations usually end. TRACER learns from real dialogues and then improves by practicing many-turn conversations to match real behaviors closely. It outperforms previous simulators, produces realistic chat flows, and helps test AI models for both conversation quality and persuasion success.
Open → 2609.28690v1

Language model outputs traced to key internal components

Matryoshka attribution: Learning to attribute language model outputs to representations and weights

Abstract: Attributing language model outputs to their internal computations is an open problem in interpretability. Existing methods, which use causal interventions, gradients, or learnable masks, either are infeasibly expensive or struggle to identify actual causally-important internal computations. We propose framing attribution as the problem of identifying nested subsets of internal components which minimise a downstream loss. To learn this task, we introduce Matryoshka Attribution (MAttr), a mask learning method that parametrises the mask with a simple differentiable sigmoid top-$k$ operator. We supervise training over all sparsities simultaneously by randomising $k$ over training, resulting in a learned ordering of components by attribution score. MAttr achieves number 1 on the official leaderboard of the Mechanistic Interpretability Benchmark (Mueller et al., 2025); our method identifies sparse and task-transferrable circuits across varying circuit bases. As a practical application, we show that MAttr can be trained with reinforcement learning to identify weight changes responsible for downstream behaviours in LLM finetuning. We train MAttr on refusal judge scores and find that restoring $1\%$ of Llama 3.1 8B Instruct's weights to their base model state is sufficient to remove refusals while maintaining capabilities. We view MAttr as a successful formulation of interpretability into a learnable objective that we can tackle with gradient descent, and encourage future work along these lines.

Tue 22 SeptComputation and LanguageMachine Learning
The gist
It is hard to figure out which parts inside large language models actually make their answers. The authors propose a new way called Matryoshka Attribution that learns to find nested groups of important components by minimizing errors. Their method can rank parts by how much they contribute and found small groups that explain model behavior well. They also showed it can identify the weight changes causing an updated model to refuse certain requests, while keeping other abilities intact.
Open → 2609.25518v1

Attention patterns reveal hallucination in large language models

Detecting Hallucination in LLMs: Tracing the Topological Signatures of Impaired Context Sharing

Abstract: In this work, we examine the topology of information flow patterns within attention graphs to effectively distinguish hallucinated from non-hallucinated responses. We analyze the Forman-Ricci curvature to identify structural patterns indicating information bottlenecks in attention graphs. We then introduce a method that captures both semi-local and global information-flow characteristics of attention heads associated with hallucinated responses. We evaluate our approach extensively across several LLMs and established benchmarks. Empirical results demonstrate that our proposed single-pass approach provides consistent improvements over existing attention-based and multi-response baselines across two hallucination-detection benchmarks, while achieving competitive performance across diverse LLM architectures. Further analysis reveals that impaired context sharing among tokens during causal generation is strongly associated with hallucination occurrences in LLMs. In particular, hallucinated responses are consistently characterized by an over-reliance on self-attention, diffused context retrieval from earlier tokens, or information over-squashing, especially in the final transformer layer.

Thu 17 SeptArtificial IntelligenceComputation and LanguageMachine Learning
The gist
Sometimes, large language models make mistakes called hallucinations, where they give wrong or made-up answers. The authors studied how these models share information inside themselves by looking at how their attention mechanisms connect words. They found that hallucinated answers show a breakdown in how the model passes context between words, especially relying too much on certain self-focused attention. This approach helps spot when the model is likely hallucinating by analyzing these connection patterns.
Open → 2609.21096v1

Dual-axis optimization method improves learning for large language model agents

Dual-Axis Policy Optimization for LLM Agents: Bayesian Feedback Attribution and Trajectory Mass Normalization

Abstract: Reinforcement learning for LLM agents involves two distinct optimization di- mensions: how environment feedback is exploited within a trajectory, and how complete trajectories are aggregated across a batch. We formulate these dimen- sions as Intra-Trajectory Feedback Attribution and Inter-Trajectory Objec- tive Aggregation, and introduce BATON (Bayesian Attribution and Trajectory Objective Normalization), a dual-axis policy optimization framework. BATON instantiates the first axis with Bayesian Feedback Attribution, which constructs a feedback-conditioned posterior over sampled actions, and the second with Trajec- tory Mass Normalization (TMN), which assigns equal optimization mass to com- plete trajectories. Experiments with GRPO and GiGPO on ALFWorld, WebShop, and SearchQA show that both axes provide independent gains and that their combi- nation consistently achieves the strongest overall performance across model scales.

Thu 17 SeptArtificial Intelligence
The gist
Training large language model agents to perform tasks based on feedback is complicated because it involves two steps: figuring out which parts of their actions led to feedback inside a single attempt, and combining results across many attempts. The authors introduce a new way called BATON that handles these two steps separately but together. One part uses Bayesian methods to better assign credit to actions inside a single attempt, while the other balances how many complete attempts are considered equally during training. Tests show that using both techniques together helps agents learn better on different tasks and models.
Open → 2609.19830v1

Suan improves safety and usefulness in large language models

Suan: Rectifying Direct Preference Safety Alignment in Large Language Models

Abstract: Integrating robust safety guardrails into Large Language Models (LLMs) is essential for delivering helpful yet harmless responses. While proprietary systems exhibit reliable safety controls, their underlying methodologies and trade-offs remain largely undisclosed. Achieving comparable security in open-weight models remains a persistent challenge, as post-trained variants frequently suffer from over-refusal and degraded general quality. To overcome these drawbacks, we introduce Suan, a novel preference optimization algorithm. Unlike existing methods, we formulate the optimization objective directly at the gradient level, bypassing the standard variational derivation. As a result, we obtain more interpretable and robust training dynamics. Extensive evaluations across a diverse suite of competitive baselines and benchmarks demonstrate that Suan achieves superior safety alignment while fully preserving response utility.

Tue 8 SeptMachine LearningArtificial Intelligence
The gist
Large language models can sometimes give unsafe or unhelpful answers, and making them safer without reducing quality is hard. The authors present Suan, a new way to train these models that directly changes how the model learns from preferences. This new method helps keep the model both safe and useful better than older techniques. They tested Suan extensively and found it outperforms other approaches on diverse safety benchmarks.
Open → 2609.08634v1