Papers for

ai model trainers

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

Phase-based reward shaping improves large language model reasoning efficiency

Phase-Conditioned Reward Shaping for Token-Efficient LLM Reasoning

Abstract: Extended reasoning can help language models solve complex problems, but long responses often contain both useful computations and redundant checks. Conventional length penalties encourage shorter responses without distinguishing the functions of intermediate steps. We propose Phase-Conditioned Reward Shaping (PCRS), a framework that combines phase identification with training-time reward allocation. We investigate two classification routes: PCRS-H with a hidden-state classifier and PCRS-T with a frozen text classifier. Our main text classifier implementation assigns overlapping reasoning, execution, and revision labels to native chunks, then uses these labels and exact repetition to adjust existing token penalties. We compare the completed configurations on six mathematical and scientific reasoning benchmarks and examine four reward-weight settings with the text classifier. Compared with a baseline using token-level length penalties, the default text classifier configuration reduces mean response length by 3.13%; sampled Pass@1 is approximately 62.07% for both methods, while oracle Pass@16 rises from 86.08% to 87.67%. These results show how phase-informed penalty allocation can reduce output volume and improve multi-attempt coverage without inference-time classification.

Sat 7 NovMachine LearningArtificial Intelligence
The gist
Long responses from language models can mix helpful calculations with repeated or unnecessary checks, making them inefficient. The authors present a way to recognize different parts of the reasoning process and adjust penalties on tokens accordingly during training. This approach helps models produce shorter, more focused answers without losing accuracy. Testing on math and science problems showed slightly shorter answers and better results when multiple attempts were allowed.
Open → 2611.11024v1

Muon optimizer makes first mixture-of-experts layer harder to balance

Muon Makes the First Mixture-of-Experts Layer Harder to Balance

Abstract: Many open mixture-of-experts (MoE) language models train with Muon instead of AdamW. Most balance the load with a bias on each expert's routing score, adjusted after every step. Their recipes disagree on whether Muon should also train the router. In models with 55M and 108M active parameters, we measure how the optimizer changes the span of these biases, the largest minus the smallest in a layer. The span grows when all tokens favor the same experts, so it shows how hard a layer is to balance. At learning rates tuned on shorter runs, Muon raised the span of the first MoE layer at every seed, roughly doubling it on average, while the other layers changed much less. The rise came from Muon on the rest of the network, whichever optimizer trained the router. In those runs, the inputs to the first router shared a direction, and soon after the warmup its busiest expert averaged four to seven times its fair share. Over the first 3,000 steps, subtracting the batch mean from these inputs removed this overload while barely changing the loss. On the smaller model, halving Muon's learning rate removed most of the rise and slightly improved the loss. Moving only the router to AdamW made the loads shift more during the warmup and slightly raised the loss, so at this scale the results favor keeping the router on Muon.

Tue 27 OctMachine LearningComputation and Language
The gist
Training large language models with mixture-of-experts layers involves balancing workload across different parts called experts. The authors found that using the Muon optimizer tends to increase imbalance in the very first layer, making some experts work much more than others. They showed that adjusting input values or lowering Muon's learning rate can reduce this problem without hurting performance. Their results suggest it’s better to keep the router training with Muon rather than switching optimizers at this scale.
Open → 2610.37914v1

Midtraining improves base ai models for software tasks and agent training

Priming Base Models for Agentic Post-Training: An Open Midtraining Recipe

Abstract: Frontier models, general-purpose and domain-specific alike, now include a midtraining stage between pretraining and post-training, but little has been published about how this stage is built. We present an open midtraining recipe that primes base models for agentic post-training, and we show its effectiveness on two Nemotron models of different sizes. Pretraining typically runs at short context and sees almost no agent interactions, while SFT data is mostly synthetic, and very long SFT stages can hurt the model's later RL trainability. Our midtraining stage sits between the two. It trains at 256K context for hundreds of billions of tokens on a diverse mixture of natural software-engineering data like packed repositories, pull requests, commit histories, and build logs together with a small share of agentic traces. On Nemotron 3.5 Lightning, 500B midtraining tokens improve SWE-bench Verified from 60.7 to 64.5, SWE-bench Multilingual from 55.8 to 62.8, and SWE-bench Pro from 32.0 to 35.5 after identical full SFT. On the larger Nemotron 3.5 Super, 280B midtraining tokens improve DeepSWE, a harder benchmark of long-horizon tasks, from 11.7 to 15.7 and SWE-bench Verified from 72.9 to 74.3 after identical full SFT. Midtraining also improves robustness across agent harnesses on both models. We report the ablations behind the recipe and an evaluation protocol built on short fixed SFT runs, since base-model proxy metrics did not correlate reliably with post-training results.

Thu 15 OctComputation and LanguageArtificial IntelligenceMachine Learning
The gist
Training large AI models in stages can affect how well they learn to do tasks. The authors show that adding a special 'midtraining' step between early training and later fine-tuning helps models better handle complex software engineering tasks and makes them more robust for agent-based training. This midtraining involves feeding models a mix of real software data and some agent interaction data at very long context lengths. Experiments on different model sizes showed clear improvements in software benchmarks and robustness after midtraining.
Open → 2610.22019v1

Language priors can override vision in multimodal AI models

Seeing is Believing: Benchmarking and Mitigating the Interference of Language Priors on Visual Perception in MLLMs

Abstract: Language priors in MLLMs can override visual evidence, impairing visual perception. Existing benchmarks assume shared priors across models and rely solely on accuracy. However, cross-model prior agreement falls below 80%, and over 15% of items can be answered correctly without any image, causing accuracy to overestimate prior resistance. To address these limitations, we introduce ViSiP-Bench, a benchmark grounded in rare-but-real scenarios across six prior categories, with a model-aware dynamic prior induction protocol and three metrics that more precisely and continuously measure prior resistance than accuracy alone. Evaluating 24 models reveals that prior adherence is pervasive, does not improve with scale, and larger models tend to either strongly override their priors or firmly adhere to them. We therefore propose MAPS, which identifies each model's strongest priors and fine-tunes on factually grounded knowledge to soften excessive prior reliance without disrupting general capabilities.

Thu 15 OctComputer Vision and Pattern Recognition
The gist
Multimodal large language models sometimes trust their language guesses more than what they actually see, leading to wrong answers about images. The authors found that models don’t always share the same language biases, and some questions can be answered right without looking at the image, making simple accuracy measures misleading. They created a new test called ViSiP-Bench with tricky examples to better measure how much models rely on language guesses versus visual evidence. They also made a method, MAPS, that fine-tunes models to reduce overreliance on these language guesses without breaking their general skills.
Open → 2610.21902v1

Multi reward training improves large language model reliability and performance

Multi-Reward Hacking: From Aggregation-Induced Drift to Trust-Anchored Defense

Abstract: To improve task performance while encouraging desirable response properties, LLM post-training often combines a primary reward (e.g., correctness) with auxiliary rewards (e.g., brevity or efficiency). However, how these rewards are aggregated can critically affect training outcomes. We show that symmetric aggregation, which treats reward channels equally, can induce a failure where aggregate reward improves while primary-task performance degrades, even when individual rewards are meaningful. We investigate this aggregation-induced failure through a one-step multi-reward decision formulation, characterizing when it constitutes multi-reward hacking. We prove that, under symmetric aggregation, deterministic conditional-mean GRPO converges to auxiliary-favoring behavior when auxiliary rewards are more reliably obtained, regardless of initialization. We propose Trust-Anchored Gating (TAG), which first elects an anchor channel based on the policy drift induced by optimizing each channel alone, then gates auxiliary credit on anchor satisfaction. Unlike calibration-sensitive static reweighting, TAG structurally eliminates credit for auxiliary-only outcomes while preserving additional credit for joint success. In the same analytical setting, TAG provably reverses this selection, converging to the primary-favoring action from any initialization when the primary channel is anchored. Experiments across Qwen and Llama models on mathematics and program synthesis reveal substantial degradation under symmetric, constrained, and normalized aggregation, whereas TAG mitigates this failure. On Qwen3-4B with a brevity reward, TAG achieves 0.581 pass@4 versus 0.220 for additive aggregation, while shortening responses by 44% relative to correctness-only training.

Thu 15 OctMachine LearningArtificial Intelligence
The gist
When training large language models, combining different goals like accuracy and short answers can sometimes make the models worse at the main task. The authors show that treating these goals equally can cause the model to focus too much on easier-to-get goals, ignoring the main one. They propose a new method called Trust-Anchored Gating that helps the model prioritize the main goal while still improving on others. Their tests show this method helps models perform better and gives shorter, more efficient responses.
Open → 2610.21404v1

Judge ensemble method improves scoring for AI training rewards

Judging the Judges: Confidence and Disagreement in Rubrics as Rewards

Abstract: Rubrics as Rewards (RaR) extends reinforcement learning to non-verifiable domains by using an LLM judge to score responses against a structured rubric. Prior work has focused on improving the rubric or strengthening the single judge, with comparatively little attention to how different single or multiple judges might be used during RaR training. We first show that judge identity alone produces a substantial spread in scores when training Qwen3-4B-Base on RaR-Medicine and RaR-Science across five frontier and open-weight judges. Building on this, we propose RaRE (Rubrics as Rewards with Ensembles), a novel judge ensemble strategy for RaR with two mechanisms: a soft-Likert reward that uses judge logprobs to weight more confident judges more heavily, and a drop-on-disagreement mask that excludes high-disagreement rollouts from the GRPO update. We find that this design is critical: while a naive average of judge scores fails to outperform the best single judge, RaRE outperforms every single-judge RaR variant on both HealthBench and GPQA Diamond, with per-theme analysis showing that it inherits the complementary strengths of its constituent judges. Our findings establish judge ensembling as a compelling axis for improving RaR.

Thu 15 OctMachine LearningArtificial IntelligenceComputation and Language
The gist
Scoring AI-generated answers is tricky because different judges often give very different scores. This paper shows that just picking one judge can cause big differences in results. The authors introduce a way to combine multiple judges into a team that weighs their confidence and ignores big disagreements. This combined approach scores better than any single judge alone on medical and science tasks, making training AI models more reliable.
Open → 2610.21179v1

Diffusion models train faster with much smaller image sets

EigDiff: Eigensample Diffusion - or how to significantly reduce training dataset size with marginal performance variation?

Abstract: Diffusion models are notoriously expensive to train, requiring large-scale datasets to learn how to gradually reverse the noise from images over multiple small denoising steps. Current approaches focus on distilling the number of diffusion steps, neglecting potential redundancies of the training data. In this paper, we explore the potential of harnessing eigensamples to train diffusion models on significantly smaller datasets, while preserving model performance. To mitigate the sensitivity to affine transformations of eigenimages, we harness the latent space of a pre-trained CLIP model. We next compute latent eigenvectors by performing principal component analysis (PCA) in CLIP space. Further, we employ eigenvectors as reference points to guide the selection of a representative training set comprising comprising both exemplars (images whose embeddings are closest to eigenvectors) and diverse images. Our comprehensive experiments with different diffusion models applied on multiple datasets indicate that our approach (EigDiff) can reduce training data by up to 14x and accelerate subset selection by up to 1607x, with marginal performance differences.

Wed 14 OctComputer Vision and Pattern RecognitionArtificial IntelligenceMachine Learning
The gist
Training computer models to generate images usually needs huge amounts of pictures and a lot of computing power. The authors found a way to pick a much smaller set of important, representative images by analyzing how pictures are grouped in a special language-image model's space. This smaller set can train the image-generating models almost as well as the full large dataset but much faster and cheaper. Their approach reduces the data size by up to 14 times and speeds up choosing the right images by over a thousand times, with very small drops in quality.
Open → 2610.20999v1

Reward models improve learning by combining specialist teachers effectively

RM-OPD: On-Policy Distillation for Generative Reward Models

Abstract: Reinforcement Learning on Mixed-domain preference data (Mix-RL) has become the mainstream approach for training general-purpose Generative Reward Models (GRMs). However, Mix-RL suffers from two critical bottlenecks: the reward sparsity induced by outcome rewards, and the cross-domain interference arising from jointly optimizing heterogeneous capabilities, which together give rise to imbalanced capability acquisition and pervasive reward hacking. Inspired by the success of On-Policy Distillation (OPD) in multi-domain capability integration, we introduce RM-OPD (Reward Modeling via On-Policy Distillation), a two-stage framework that separates specialist acquisition from capability integration. First, we cultivate domain-specialized teacher models via single-domain GRPO fine-tuning, allowing each expert to reach its performance ceiling in isolation; then we consolidate these teachers into one student model through (i) routed on-policy distillation, which provides dense signed token-level supervision on student-generated critiques; (ii) domain contribution reweighting, which reweights token guidance toward target domain shares, preventing specific domains from dominating optimization; and (iii) optional criteria context distillation, which conditions teachers on domain-specific evaluation criteria to sharpen token-level supervision. Across multiple RM benchmarks and two base models, RM-OPD consistently outperforms vanilla Mix-RL and other strong baselines, achieving overall superior performance and exhibiting an emergent "teacher-surpassing" effect. These results establish RM-OPD as a promising paradigm for building generalist GRMs.

Wed 14 OctMachine LearningComputation and Language
The gist
Training AI models to understand human preferences from mixed kinds of data is hard because different task types confuse the model and reward signals can be very sparse. The authors show a way to train separate specialized models for each task first, then combine their knowledge into a single model. They do this by teaching the combined model directly using detailed feedback from the specialists and controlling how much each specialist influences the combined model. This method beats previous approaches and sometimes the combined model gets even better than its teachers.
Open → 2610.20174v1

Post-training helps models focus compute on hardest reasoning problems

Beyond Correctness: Compute-Responsive Data Selection for Reasoning Model Post-Training

Abstract: Reasoning language models increasingly rely on test-time compute scaling, yet most existing work studies how to allocate inference computation while treating the model itself as fixed. We ask a complementary question: can reasoning models be post-trained to make better use of a given amount of test-time compute, and which examples are most useful for this goal? Our key observation is that reasoning problems differ substantially in their response to additional computation. Some are already solved under limited compute, others remain unresolved even after further reasoning, while a distinct subset changes from complete initial failure to success when additional reasoning is provided. We call these compute-responsive (CR) examples and introduce Compute-Responsive Data Selection (CRDS), a post-training data-selection principle that prioritizes examples with empirically demonstrated marginal utility from additional reasoning. On Qwen3-8B, SFT on CR examples recovers 45.3\% of held-out low-compute failures, compared with 40.6\% when training on already-solved examples. The advantage extends beyond the selected CR split: SFT-CR reaches 51.8\% overall accuracy on an independently sampled test distribution, with the gain concentrated on compute-responsive problems. CR training also shifts the compute--accuracy frontier across inference budgets from 512 to 4,096 tokens, outperforming matched non-CR training throughout and reaching 48.1\% recovery at the largest budget. The same pattern is supported across multiple data sources, remains visible under on-policy distillation, and extends to Llama-3.1-8B-Instruct when CR sets are reconstructed separately for that model. Better test-time compute utilization can therefore be learned through post-training, with recoverable responses to additional reasoning providing a particularly informative signal for selecting training examples.

Wed 14 OctArtificial Intelligence
The gist
Reasoning models get better when they use more compute, but not all problems improve equally with extra effort. The authors found that some reasoning tasks only succeed when the model spends more compute — these are called compute-responsive examples. By retraining models to focus on these compute-responsive examples, they make better use of available compute during testing, solving harder problems more reliably. This approach improved performance on multiple models and datasets without needing model changes during inference.
Open → 2610.20057v1

Larger batches slow learning but predict failure timing in reinforcement learning

Two Clocks: Why Larger Rollout Batches Learn Slower but Fail on Schedule in Reinforcement Learning with Verifiable Rewards

Abstract: Reinforcement learning with verifiable rewards (RLVR) relies on costly model-generated responses, motivating larger rollout batches and square-root learning-rate scaling. Reported to approximately preserve progress per sample with one optimizer step per rollout, this rule is less understood in practical multi-step training. We develop a two-clock framework that separates the samples needed to make progress from those consumed before collapse. Under a local quadratic model, the square-root rule preserves first-order drift per sample but not progress: larger batches take fewer, longer updates and lose more to curvature, yielding a learning-rate-dependent critical batch beyond which sample efficiency degrades. A drift-threshold hypothesis predicts a complementary failure clock whose sample count remains approximately invariant along the square-root rule. Controlled GRPO experiments on Qwen2.5-Math-1.5B across a $16\times$ batch range support both predictions. Ratio clipping delays failure, especially for larger batches, but the progress gap persists without it. Widening the clip trades stability for a transient speed gain. The progress gap reappears on another dataset, a general-purpose checkpoint, and a 7B model. Finally, we use the two-clock framework to derive a simple multi-step RLVR schedule: start below the critical batch and switch to a larger batch before failure. Our code is available at: https://github.com/WinstonLiyt/two-clocks-rlvr.

Tue 13 OctMachine LearningArtificial Intelligence
The gist
Getting a computer to learn by trying many things and checking rewards can be expensive when using big batches of examples at once. The authors found that using bigger batches slows down learning because updates become less efficient due to complex effects in the math. They introduce a 'two-clock' idea to explain how progress and failure happen on separate timescales and show that adjusting batch sizes over time can improve learning. Their experiments with large language models support these ideas and suggest practical ways to schedule learning.
Open → 2610.18853v1

Proximal KL improves off-policy reinforcement learning for math tasks

Enabling Token-wise Importance Sampling in Off-Policy RLVR with Proximal KL Regularization

Abstract: Off-policy data in reinforcement learning with verifiable rewards (RLVR) can arise from asynchronous generation, rollout-training mismatch, and response reuse. Token-wise importance weighting avoids multiplying likelihood ratios across a response but generally biases policy gradients. We prove that, under coverage and local expressiveness conditions, full future KL credit makes the exact proximal optimizer the update's unique stationary policy, even for mixtures of older policies, without full-response weights or a fitted critic. We further establish convergence of continuous-time tabular population updates under persistent coverage and time-varying behavior policies, and a finite-sample reward guarantee for linear logistic models under additional assumptions. These results motivate Proximal KL, a practical off-policy RLVR method that separates the proximal reference from the sampling policy and replaces PPO clipping with bounded future KL credit. Across all seven matched off-policy comparisons on Qwen3-1.7B and Qwen3-4B, spanning two learning rates, response reuse, and rollout staleness, Proximal KL outperforms GRPO in micro-averaged accuracy over eight math benchmarks by up to 2.58%. A separate on-policy comparison yields comparable aggregate accuracy. In periodic-refresh experiments, Proximal KL avoids the collapses observed in PG and CISPO, exceeds GRPO's training accuracy, and reaches higher accuracy in fewer policy updates. Ablations show that token-wise correction outperforms sequence-level and unweighted controls. With token-wise correction retained, future KL feedback reduces sensitivity to the tested baseline perturbation relative to PPO clipping. A finite future horizon improves AIME24 mean@16 and pass@16 over the full tail, while fixed-policy measurements show that truncation reduces correction variance over the tested horizons.

Tue 13 OctMachine Learning
The gist
Reinforcement learning helps computers learn by trial and error, but using old or mismatched data can make this learning unstable or biased. The authors propose a new method called Proximal KL that better accounts for each piece of data separately, preventing mistakes that happen when combining data improperly. They show mathematically that this approach converges well and performs better than existing methods on math-focused language models. Their method keeps training stable and improves accuracy in fewer steps.
Open → 2610.18738v1

Self-distillation shows limited reasoning gains across new problems

Does self-distillation improve reasoning?

Abstract: Self-distillation is an emerging approach that trains a model to imitate its own predictions when conditioned on privileged information, such as correct answers or environmental feedback. Recent methods have reported better sample efficiency and generalization than reinforcement learning (RL), but it remains unclear whether these gains extend beyond in-distribution evaluations and whether they stem from privileged distillation. In this paper, we revisit two representative methods, Self-Distillation Policy Optimization (SDPO) and On-Policy Self-Distillation (OPSD), across code, science, tool use, and math reasoning tasks. We find that their reported gains do not consistently generalize to unseen problems and are largely explained by overfitting or transferring existing reasoning behavior, rather than effectively using privileged information. Moreover, self-distillation gradients show weak alignment with policy gradients that directly optimize expected reward, questioning whether they provide an effective direction for policy improvement. Our experiments across Qwen3 models (1.7B-14B) and OLMo-3-7B suggest that effectively leveraging privileged information to improve reasoning remains an open challenge. Our code is available at https://github.com/sleepypandu/does-sd-improve-reasoning

Tue 13 OctMachine LearningArtificial IntelligenceComputation and Language
The gist
Self-distillation is a way to train AI models by making them copy their own improved answers when given extra helpful information. The authors tested this method on tasks like coding, science, and math to see if it helps AI solve new problems better. They found that it usually doesn’t improve AI’s ability to reason or generalize beyond what it already knows. Instead, the gains come from memorizing or repeating existing knowledge, and the training signals don’t align well with actually improving the AI’s success. This means making AI reason better with this technique remains a challenge.
Open → 2610.18613v1

Replay improves rubric-based reinforcement learning with adaptive teacher hand-off

RELAY: Selective Self-Distillation with Adaptive Hand-off for Rubric-Based Reinforcement Learning

Abstract: Rubric-based reinforcement learning grades a response against many written criteria but collapses them into one reward, which discards the detail the rubric carries and gives no GRPO reward gradient when a group's responses tie. A frozen teacher that reads the rubric as privileged context provides token-level supervision, but we observe an early plateau in the gains over GRPO. RELAY uses advantages and rewards to select positive-advantage and low-scoring zero-advantage rollouts, and monitors the distillation loss to decide when to retire the teacher. Stage 1 combines selective distillation from a frozen teacher with GRPO; a plateau detector then removes the teacher, and Stage 2 continues with GRPO alone. Calibrated on one model-domain pair and reused unchanged in three others, the plateau detector fires between steps 122 and 171. RELAY scores highest in-domain among the compared methods in all four settings, ahead of GRPO on RaR-Medicine by 4.62 points with Qwen3-1.7B and 6.02 with Qwen3-4B. On Qwen3-1.7B/Medicine, removing the teacher from a shared checkpoint outperforms keeping, weakening or refreshing it. Distilling selected rollouts covering about half the response tokens also outperforms distilling all rollouts at a matched average coefficient.

Tue 13 OctMachine LearningArtificial IntelligenceComputation and Language
The gist
Grading responses with many criteria often loses detail when combined into one score, making learning less precise. The authors introduce RELAY, a method that uses a smart system to selectively learn from a fixed 'teacher' model that understands the grading details, then switches to learn independently when progress slows. This approach improves performance across multiple test settings, scoring higher than previous methods. RELAY also shows better results by only learning from selected useful examples instead of all data.
Open → 2610.17623v1

Language models learn from each other to improve reasoning skills

JustOPD: On-Policy Distillation from Autoregressive to Diffusion Language Models

Abstract: On-policy distillation (OPD) is a post-training approach for large language models (LLMs) that enables a student model to learn a teacher model's distribution from sequences sampled from the student's own policy. Despite its promising capability, OPD for diffusion language models (DLMs) remains underexplored and existing studies primarily focuses on DLM-to-DLM distillation. The limited availability of strong teacher DLMs and the flexible generation order of DLMs impose additional technical challenges as well. However, it remains unclear how to effectively distill knowledge from a strong autoregressive language model (ARM) into a weaker DLM, given the mismatch between the left-to-right decoding of ARMs and the flexible decoding order of DLMs. Inspired by Ni et al. (2026), we propose JustOPD, an ARM-to-DLM on-policy distillation method that uses autoregressive rollouts during training to align the student's generation process with the ARM teacher's left-to-right decoding. This alignment enables effective distillation from ARM teachers while preserving the DLM's flexible generation-order capability at inference time. Experiments using Qwen3-14B as the teacher and Dream-7B as the student demonstrate that JustOPD consistently improves mathematical reasoning performance over the base DLM across multiple divergence choices.

Mon 12 OctMachine Learning
The gist
Large language models can teach smaller ones to perform better by learning from the smaller model's own generated text. The authors found a new way to help a flexible model learn from a stronger model that reads text left to right, even though flexible models generate text in a different order. Their method lets the new model keep its unique style while improving its math skills. Tests showed the improved model solved math problems better than before.
Open → 2610.17397v1

Composing coding tasks boosts benchmarks and model training quality

CodeWalk: Generating Coding Benchmarks by Walking a Problem Graph

Abstract: High-quality coding problems are expensive to collect. Benchmarks such as SWE-bench rely on substantial human curation, while synthetic alternatives typically rely on costly model generation and validation. In this work, we show that new coding problems can be created at almost no cost by composing existing ones. We formulate problem generation as code composition: verified atomic problems are combined into level-$N$ tasks, and a task is kept only if every atomic problem remains unsolved and a joint reference solution passes the same check used at evaluation time. The entire task-construction process is programmatic and requires no language-model generation or validation. We instantiate this approach in four coding settings: multi-bug repair, proof completion, complexity analysis, and clone equivalence. The generated tasks are challenging for frontier models, which all degrade as compositional depth increases. The generated tasks are also useful as training data: rejection-sampling fine-tuning on them improves Qwen3.5-9B at every composition level.

Sun 11 OctMachine LearningArtificial Intelligence
The gist
Collecting good coding problems is usually costly because it requires a lot of human work or complicated checking. The authors show that new, hard coding tasks can be created cheaply by combining simpler, verified problems in a careful way. They test this in different coding challenges and find that harder combined tasks give tougher tests for AI models and can help improve their training. This process doesn’t need expensive model-generated content or extra validation steps.
Open → 2610.15345v1

Spectral weight decay improves neural network compression and accuracy

Spectral Weight Decay: Inducing Low-Rank Structure in Neural Network Weights

Abstract: Standard weight decay treats each weight matrix as a vector and ignores its spectral structure. We introduce spectral weight decay, a post-step decoupled nuclear-norm update that applies additive rather than multiplicative spectral shrinkage. We connect the update to approximate proximal descent and show that its sensitivity to update order can exceed that of conventional $\ell_2$ weight decay near rank deficiency. Across LLaMA models with $124$M to $500$M parameters, spectral weight decay lowers effective rank and improves SVD-LLM compression at matched validation loss. At $500$M and a $4\%$ distortion budget, it reaches $1.89\times$ compression and $1.18\times$ GPU inference speedup, compared with $1.14\times$ and $1.01\times$ after standard weight decay. Under fixed-horizon training with $60\%$ label noise, it also improves final mean clean-test accuracy over matched $\ell_2$ regularization by up to $17.8$ points on MNIST and $4.6$ points across four BERT-base tasks. Code is available at https://github.com/brain-lab-research/SpectralWD.

Thu 8 OctMachine Learning
The gist
Neural networks learn by adjusting many numbers called weights, but these weights often have hidden patterns that regular methods ignore. The authors introduce a new way to gently shrink these weights by focusing on their underlying structure, making the network simpler without losing accuracy. This method helps compress the model, speeding up its use, and improves how well networks learn from noisy data. They tested this on popular language and vision models and saw clear benefits.
Open → 2610.11730v1

Flatness alone does not drive better generalization in neural nets

Do Flatter Minima Drive Better Generalization? An Algorithmic Separation in Grokking

Abstract: Flat loss landscapes have long been linked to better generalization in neural networks. However, its role as a causal mechanism for generalization is less established. Grokking provides an unique testbed to understand this distinction: models are prone to fit observed data using non-generalizing structure and remain in that regime for prolonged periods, transitioning to generalization only under particular training conditions. In this work, we study whether flat loss landscapes can act as a driving mechanism in this transition. While recent work has argued for flatness as a necessary geometric condition for this transition, we find that biasing training toward flatter solutions using sharpness-aware minimization (SAM) is insufficient to reliably induce this transition, despite producing flatter solutions. However, when SAM is paired with mechanisms that drive generalization such as weight decay, an interesting property emerges: SAM can accelerate the transition to generalizing solutions by up to 4x at the epoch-level. We theoretically untangle this relationship between SAM and weight decay using a minimal interpolating two-layer ReLU model with both memorizing and generalizing solutions. We show that even in this simple setup, flatness alone cannot distinguish a memorizing solution from a generalizing one, while weight decay favors generalizing solutions. However, under a local stability analysis, there exists a window where a memorizing interpolant is locally stable under gradient descent but unstable under SAM in the low-norm regime, which can explain SAM's ability to accelerate this transition. Overall, our results provide a more interpretable account of the role of flatness in driving generalization, especially in settings where models are vulnerable to minimizing loss through learning non-generalizing structure.

Thu 8 OctMachine Learning
The gist
Knowing whether a computer model that is good at remembering data will also be good at learning general patterns is hard. The authors study how ‘flatness’ of the training error impacts this. They find that just making a model's training landscape flatter does not guarantee it will learn generalizable rules. However, when combined with techniques like weight decay, flatter solutions can speed up learning of generalizable patterns. Their math model shows how flatness and weight decay interact, clarifying when and why flatness helps.
Open → 2610.11206v1

Large models resist early training biases better than small models

Sequential Pretraining Favors Large Models

Abstract: Large neural networks often acquire capabilities that small models fail to learn. Does this stem from large models learning more representative features, or from being more robust to unaccounted-for adverse effects introduced during training? We define and quantify one such adverse effect, primacy bias, as the extent to which exposure to early data distributions impairs later learning. We show that small models can allocate learning capacity inefficiently toward early distributions, whereas sufficiently overparameterized models are robust to this effect. This inefficiency is particularly consequential in pretraining, where foundation models often encounter heterogeneous data distributions sequentially rather than jointly. As a result, small foundation models can struggle to learn distributions encountered late in training, which is particularly harmful when later data emphasizes desirable capabilities such as code, mathematics, and reasoning. Motivated by these findings, we introduce Exposure Therapy (ET), a simple regularization that promotes more efficient allocation of learning capacity during sequential pretraining. We demonstrate that ET improves foundation models' performance on late data distributions as well as overall capability in models up to the billion-parameter scale. Overall, our results suggest that some benefits of large foundation models may arise from greater robustness to adverse training effects, rather than from learning more representative features, and that improved training algorithms can recover some of these advantages in smaller models.

Wed 7 OctMachine Learning
The gist
When training AI models on different types of data one after another, small models tend to focus too much on the earlier data and struggle to learn from later data that may be more important. The authors show that bigger models handle this problem better because they can allocate their learning more efficiently. They introduce a method called Exposure Therapy that helps smaller models learn from later data better by reducing this early-data bias. Their experiments show that this method improves the overall ability of smaller models to learn complex tasks.
Open → 2610.09611v1

Klpo improves policy updates for large language model training

On KL-Regularized Policy Optimization

Abstract: Asynchronous reinforcement learning (RL) for large language model (LLM) agents trains one policy on trajectories generated by another: rollouts come from stale checkpoints, and the inference engine's probabilities differ from the trainer's even at identical parameters. Standard remedies either clip importance ratios, which biases the update, or, as in GRPO, sample a group of responses per prompt, which is costly when episodes are long. We propose KL-Regularized Policy Optimization (KLPO), a framework that anchors the KL regularizer at the sampler. The regularized improvement step then has a closed-form Gibbs solution, and KLPO fits its log-ratio optimality condition by least squares on the sampler's own trajectories, so the sampler probability enters through a log-ratio and no importance weights are needed. Profiling out the regression intercept replaces the intractable log-partition function with the signal's sampler mean plus a sampler-to-trainer KL divergence. For token-level policy mirror descent targets, we show that the resulting gradient can be computed from terminal returns without a critic, via sampler-centered scores or a single trajectory residual, even under stochastic tool outputs. We further prove that independent Monte Carlo estimates of the KL term keep these gradients unbiased, derive the exact KL gap of cheaper top-$K$ and binary approximations, and show that SPPO, GPO, REBEL, and BPO arise as special cases of KLPO. The result is a critic-free update that uses one rollout per prompt and requires neither a learned normalizer nor a group of responses.

Tue 6 OctMachine LearningArtificial IntelligenceComputation and Language
The gist
Training AI agents like large language models often involves using older versions of the model to generate examples, which can cause problems for learning updates. The authors propose a new method called KL-Regularized Policy Optimization (KLPO) that avoids common issues by tightly connecting the training and sampling processes. KLPO makes policy updates without needing extra complex calculations or multiple outputs per prompt, making training simpler and more efficient. It also unifies several existing methods under one framework and works without needing additional prediction tools.
Open → 2610.08963v1

Language models sometimes ignore their own moral judgment under pressure

Principled Under Pressure: Post-Training Decides Whether LLMs Act on Their Own Moral Judgment

Abstract: Language models increasingly act as agents. An agent that says an action is wrong and then takes it anyway is a different failure from one that does not know better, and evaluations of stated values cannot see it. We build a pre-registered panel of 248 scenarios across five kinds of pressure. Each scenario is posed twice to the same model, once as the agent choosing what to do and once in the third person asking which option is right, so the model's own judgment is the reference. Every scenario has a twin with the pressure removed, and every model gets a positive control in which its operator orders the violating action, so that a missing gap can be told apart from a blind instrument. On OLMo-3-7B-Instruct, the model takes the action it judged wrong on about one in five pressuring scenarios, more often than on the same scenarios with the pressure removed. Across four instruct models the gap depends on the post-training recipe: OLMo-3 and Meta's Llama-3.1-8B-Instruct carry it; Tulu 3 shows none on the whole panel (above about 0.01 in probability) or on its own most-pressuring scenarios; Qwen2.5-7B-Instruct shows none on the whole panel (above about 0.02) and is unresolved on its own (0.083, -0.028 to 0.195). Meta's recipe and Ai2's Tulu 3 start from the same Llama-3.1 weights, and only Meta's carries the gap. Reading a chat model outside its chat template reverses the sign of its gap with nothing at stake (-0.038 against +0.055 under the template on OLMo-3), a distortion present on two of three recipes. On both models that carry it, reasoning about the stakes before acting moves the choice back toward the model's own judgment, against a same-length non-moral task, with or without the pressure; on OLMo-3, naming the norm at stake does about a third of that. The gap is a measurable target for post-training recipes, not a fixed property of pretrained weights.

Tue 6 OctMachine LearningArtificial IntelligenceComputation and Language
The gist
This paper shows that language models acting like agents can say an action is wrong but still do it when pressured. The researchers tested models with scenarios that had pressure on them and found some models broke their own moral rules more under pressure. Different post-training methods affected whether models stuck to their moral judgment or not. Thinking about consequences or naming the moral rule helped reduce this gap. These findings mean how models are trained after initial learning matters for whether they act consistently with their stated values.
Open → 2610.08670v1

Small language models learn better by mixing teacher advice based on confidence

Learning What to Imitate: Entropy-Aware Distribution Mixing

Abstract: Small language models are often post-trained as students on reasoning traces from stronger teacher models to efficiently learn new skills. However, token-level imitation on traces that lie far outside the student's expected distribution often produces \textit{confident conflicts}, whereby the student is required to imitate a continuation that it deems unlikely (i.e., low-probability) despite being confident in a different continuation (i.e., in a low-entropy state). To mitigate the degradation in generalisation and catastrophic forgetting caused by these conflicts, we propose \textbf{Entropy-Aware Mixing}: a dynamic per-token interpolation of the student and teacher distributions, gated by the student's predictive entropy. We implement both convex and geometric interpolations for both offline trace generation (via speculative decoding, then SFT) and on-policy forward-KL distillation. Our results show that entropy-aware mixing stabilises distillation, improving in-distribution and out-of-distribution math reasoning while better preserving general capabilities than fixed-teacher supervision. Nonetheless, the optimal entropy schedule depends on the training source, with offline-generated traces favouring concave schedules (greater overall teacher influence) and on-policy training favouring linear or convex schedules (teacher concentrated in high-entropy states).

Mon 5 OctMachine Learning
The gist
When smaller AI language models learn by copying bigger, smarter models, they sometimes get confused because the examples they try to copy seem unlikely to them. This causes problems in learning and remembering skills. The authors propose a new way to blend what the small model believes with the teacher's advice, adjusting this blend depending on how confident the smaller model is at each step. This approach helps make learning more stable and improves the smaller model’s reasoning abilities without losing its general skills.
Open → 2610.06671v1

Language models spot impossible problems but wrongly say solved

Language models can notice an impossible engineering problem yet still report it as solved

Abstract: Language models draft engineering calculations, but answer accuracy does not show whether they reject an impossible problem. We tested 14 models on 30 pairs of mechanics problems, each with a valid version and one made impossible by changing a given value or assumption. Two independent solvers verified every answer key and showed that each flawed problem was physically impossible. We scored solving of valid problems separately from rejection of their flawed counterparts. Each reply required a "solved" or "cannot solve" status; rejection meant "cannot solve" or withholding an answer. The initial prompts did not warn that problems could be flawed. Across three recent models, 12 of 90 replies failed to reject a flawed problem. In 11 of these replies, the model stated the flaw, answered a corrected problem and still reported the original as "solved", according to artificial intelligence raters and numerical checks. We later retested four models from one provider, offering "flawed" instead of "cannot solve" and asking them to name and explain the defect. Three models showed statistically significant increases in rejection, but valid-problem solving fell in three. Evaluations therefore need to score both versions and distinguish flaw recognition from the reported status.

Mon 5 OctArtificial IntelligenceComputational Engineering, Finance, and ScienceComputation and Language
The gist
Sometimes language models can recognize when an engineering problem is impossible to solve, but they still say they solved it. The researchers tested several models with pairs of mechanics problems—one valid and one impossible. Even when models pointed out the issue, many still marked the problem as solved. Changing the instructions helped models reject flawed problems more often, but it also reduced their success on valid ones.
Open → 2610.06668v1

Dynamic resolution helps improve multimodal AI debates accuracy

DREAM: Dynamic Resolution Assignment For Multimodal Multi-agent Debate

Abstract: Multi-agent debate (MAD) has emerged as an effective paradigm to improve the reasoning capabilities of large language models (LLMs) and is increasingly being extended to multimodal settings. However, existing multimodal MAD frameworks typically expose agents to the same fixed visual input, ignoring substantial variation in the visual scale needed across samples and agents. In addition, these frameworks frequently suffer from groupthink, a phenomenon where agents prematurely abandon correct deductions to conform with confident but hallucinated peer responses. To address these bottlenecks, we introduce DREAM (Dynamic Resolution Assignment For Multimodal Multi-Agent Debate), which operates via two core components: (1) Dynamic Resolution Assignment, a zero-shot probe round where agents test multiple resolutions, quantify uncertainty using Average Normalized Log-Likelihood (ANLL), and use an adaptive threshold to assign each agent to its empirically optimal resolution; (2) Uncertainty-Guided Rollback Aggregation counters groupthink by tracking each agent's uncertainty over rounds and restoring early low-uncertainty answers overridden by group pressure. On six multimodal datasets, DREAM improves the accuracy-token trade-off over multi-agent debate baselines by 1.5-3.2% accuracy without dataset-specific tuning.

Sun 4 OctArtificial IntelligenceComputation and LanguageComputer Vision and Pattern Recognition
The gist
It can be hard for AI models to look at pictures and text together and make good decisions because they usually see the same visual details, even when different parts need more or less focus. The authors designed a method where AI agents first try out different image resolutions to find the best detail level for each agent to work with. They also have a way to prevent agents from copying each other’s mistakes by tracking how sure each agent is about their answers over time. This approach made the AI teams more accurate without needing special tuning for each dataset.
Open → 2610.05615v1

Scaling task updates improves learning from clustered training data

Task Vector Descent: Learning from Non-IID Batches

Abstract: A central challenge in continual learning is to acquire new knowledge without forgetting what the model has already learned. This challenge appears in language model training when training data comes from various domain-, user-, or task-specific distributions that are encountered unevenly over time. In such settings, successive minibatches are temporally clustered by distribution instead of being sampled i.i.d. from the overall data mixture. Training on temporally clustered data induces a stability-plasticity tradeoff. Adapting the model to the active distribution can improve the model on the active distribution but may lead to a performance degradation on data it previously trained on. We find that this tradeoff intensifies with longer exposure to the same distribution. We therefore ask if the parameter displacement produced by such a sequence (the task vector) should be fully retained or applied partially. We compare applying the full displacement ($λ=1$) with partial integration, which scales the task vector by $λ$ before applying it to the continuing model and scales the optimizer state by the same coefficient. Across continual pretraining, pretraining from random initialization, supervised post-training, and reinforcement post-training, we find that intermediate values of $λ$ often improve average continuing-model performance relative to full integration, particularly after longer same-distribution sequences. In continual-pretraining experiments with both controlled streams and naturally defined adaptation sequences, task-vector scaling outperforms full integration at the matched learning rate, showing that its benefits are not reproduced by learning-rate scaling alone.

Sun 4 OctMachine LearningComputation and Language
The gist
Models that learn from data coming in groups focused on one topic often struggle to remember older knowledge while adapting to new information. The authors studied if applying all updates from one topic at once or only part of them works better. They found that using only a fraction of these updates often helps the model keep a better balance between learning new things and retaining old knowledge. This approach was tested in various learning scenarios and showed improvement beyond just tweaking the learning rate.
Open → 2610.05402v1

Topological method improves concept editing in neural networks

Operational Abstractions of Neural Network Concepts via Topological Representations

Abstract: Concept-based methods provide a semantic level for interpreting and manipulating learned representations, but existing editing approaches are typically specialized to particular interventions and do not provide a common and editable representation of concept organization. To achieve this, we introduce Topological Concept Representations (TCR), a post-hoc operational abstraction that jointly characterizes the concepts encoded in a learned representation and their relationships. TCR constructs an intermediate concept space from concept recoverability and interaction scores, and compactly encodes its organization through topology. Interventions are expressed through modifications of this abstraction that are propagated back to the underlying learned representation. This allows different concept-level operations to share the same optimization framework and separates the desired concept organization from the mechanism to achieve it. We establish stability and reparameterization-invariance properties of TCR and its connections to existing concept-editing formulations. We use TCR to disentangle concepts as a preprocessing step for existing erasure methods, improving worst-group accuracy by 21.89 on average at comparable concept leakage. We further use TCR to transfer concepts from teacher to student models, improving concept recoverability by up to 5.54 while also improving or maintaining competitive test top-1 accuracy.

Sun 4 OctMachine LearningArtificial Intelligence
The gist
Neural networks learn many hidden concepts, but changing these concepts inside the models is tricky and usually done in very specific ways. The authors propose a new approach called Topological Concept Representations that maps these concepts and their relationships into a simpler, organized form. This makes it easier to edit and transfer concepts across models while keeping performance stable. They show this method helps improve accuracy and better control of concepts in neural networks.
Open → 2610.05017v1

Sequential reward scheduling improves text to 3D model training

OuroReward: Sequential Reward Scheduling for Reinforcement Learning in Text-to-3D Generation

Abstract: Reinforcement learning (RL) for Text-to-3D (T23D) generation requires optimization across multiple quality dimensions such as semantic alignment and texture clarity. Existing methods typically optimize these dimensions simultaneously through multiple reward aggregation, without explicitly modeling inter-dimension dependencies. This can cause imbalanced optimization and persistent interference among conflicting dimensions. To address this limitation, we propose OuroReward, an interference-aware sequential reward scheduling strategy for T23D RL. OuroReward first estimates pairwise dependencies among dimensions and constructs a cyclic optimization path that minimizes cumulative interference. By incorporating the tail-to-head dependency, the cycle captures global compatibility across the entire schedule. Then, OuroReward converts the cycle into a one-pass sequence, and starts optimization from the dimension with the lowest aggregate interference. Rather than assigning a fixed optimization budget to each dimension-wise reward, training adaptively determines when to advance to the next reward according to the remaining optimization headroom of the current one. We further introduce AdaSelect, an adaptive prompt selection strategy that identifies reliable and informative prompts aligned with the model's current capability. By focusing policy updates on these prompts, AdaSelect effectively improves training stability. Extensive experiments across different T23D models and RL algorithms demonstrate that our framework consistently improves generation quality across multiple dimensions.

Fri 2 OctComputer Vision and Pattern Recognition
The gist
Making 3D models from text descriptions requires balancing different qualities like matching the description and having clear textures. The authors noticed that fixing all qualities at once can cause conflicts and poor results. They created OuroReward, a method that plans the order of focusing on each quality based on how they affect each other to reduce interference during learning. They also developed AdaSelect, which picks good example descriptions to train on, helping the learning process stay stable. Their tests show these ideas improve the quality of 3D models created from text.
Open → 2610.03423v1

On policy distillation focuses teacher tokens on key successful and failed examples

Spend Teacher Tokens Where They Matter: Success-Referenced On-Policy Distillation

Abstract: On-policy distillation (OPD) combines student-generated rollouts with dense token-level supervision from a teacher, but providing such supervision for every rollout requires substantial teacher computation. We introduce Success-Referenced On-Policy Distillation (SR-OPD), which reduces this cost by selecting which prompts and rollouts receive teacher supervision. When the student produces both successful and failed rollouts for the same prompt, a successful rollout can serve as a natural reference for selecting failed rollouts. SR-OPD therefore focuses on such prompts and prioritizes failed rollouts whose hidden-state trajectories show sustained divergence from a successful reference, while accounting for estimated teacher-input cost. Across three teacher-student pairs and six mathematical reasoning benchmarks, SR-OPD uses only 3.46-5.02% of the teacher-input tokens required by Vanilla OPD in the one-pass setting while maintaining comparable reasoning performance. Under a controlled setting matched to 5% of Vanilla OPD's teacher-input budget, further experiments support both key design choices: focusing supervision on prompts with both successful and failed rollouts, and using successful rollouts to guide failure selection. These results indicate that a student's own successful behavior can serve as a useful reference for allocating teacher supervision under a fixed teacher-input budget.

Fri 2 OctArtificial Intelligence
The gist
Teaching large AI models can be very costly because it usually requires checking every example they try and correcting them. The authors propose a smarter method where the AI learns mostly from its mistakes where there is also a successful attempt to compare against. This way, the teacher AI only needs to spend its effort where it really helps, saving a lot of work without hurting the AI’s learning. Their method uses just a small fraction of teaching tokens but still keeps the AI’s problem-solving skills high.
Open → 2610.02678v1

Cancellation-aware masking improves reinforcement learning in large language models

CARM: Cancellation-Aware Response Masking for LLM Reinforcement Learning

Abstract: Recent years have witnessed the rapid adoption of reinforcement learning (RL) in large language model (LLM) post-training, with substantial gains in mathematical reasoning and code generation. In practical systems, however, policy updates and differences between rollout and training engines can make sampled responses off-policy. Sequence-level masking addresses this mismatch by deciding whether an entire response should contribute to optimization. A common masking rule uses the length-normalized geometric mean of sampled token probability ratios. Its signed log-ratios can cancel across positions, concealing substantial bidirectional policy drift. We propose \emph{Cancellation-Aware Response Masking} (CARM), a sequence-level mask that takes the absolute value of each token log-ratio before averaging, preventing opposing probability changes from canceling. We prove that accepted responses satisfy a joint bound on the fraction of sampled-token ratios outside a prescribed band and their mean log-distance beyond its boundaries. Experiments on mathematical reasoning and code generation show that CARM improves mean@16 averaged over AIME 2024/2025/2026 and BeyondAIME by up to $3.13$ percentage points over geometric-mean masking, and increases average pass@1 across four code benchmarks by $2.88$ points over the strongest evaluated baseline. These findings support CARM as a theoretically grounded and effective method for response-level off-policy control in LLM reinforcement learning.

Thu 1 OctMachine LearningArtificial IntelligenceComputation and Language
The gist
Training large language models with reinforcement learning can be tricky because updates sometimes conflict with the way responses are generated. The authors found that the usual method for deciding which responses to learn from can miss important differences due to balancing effects in the math. They created a new masking method called CARM that catches these hidden differences better by looking at token changes without letting opposing effects cancel out. CARM helps the model learn more effectively and improves performance on math problems and code writing.
Open → 2610.02039v1

Asynchronous learning method improves language model training speed and stability

Asynchronous LLM Post-Training: Group-Mass Capping and Convergence Analysis

Abstract: Asynchronous reinforcement learning (RL) improves the efficiency of large language model post-training but introduces stale rollouts generated by earlier policies. Theoretical understanding of how this staleness affects convergence and how to mitigate its impact remains limited. We derive a convergence bound for GRPO-style algorithms that explicitly characterizes the tradeoff between the gradient estimator's second moment and bias. For trajectory-level importance-weighted estimators, our analysis shows that once the second moment is uniformly controlled, delay enters the bound through the bias introduced by clipping or rescaling. Guided by this insight, we propose a novel group mass capping GRPO (GMC-GRPO) method, which minimizes a ratio-based bias bound within a class of weighted estimators sharing a common second-moment guarantee. We establish convergence guarantees for asynchronous GMC-GRPO and show that, compared with TIC-GRPO, it improves the threshold dependence of the fourth-order delay term from $O(ε^{-4})$ to $O(ε^{-2})$ as $ε\to0$, where $1+ε$ is the ratio threshold. Under local policy overlap, the delay-dependent term decreases as $G^{-2/5}$ after tuning the step size, where $G$ is the group size. For fixed behavior and current policies, the bias introduced by group rescaling also vanishes as $G\to\infty$, whereas the bias from trajectory-wise clipping can persist. Experiments across Qwen3 models and reasoning benchmarks demonstrate improved robustness to stale rollouts, with GMC-GRPO achieving the best performance among stable baselines under large rollout delays.

Thu 1 OctMachine LearningArtificial Intelligence
The gist
Training large language models using asynchronous methods can speed up learning but may cause problems because the model updates use outdated information. The authors study how this delay affects the training process and introduce a new technique, called group mass capping, that reduces the negative effects of these delays. Their method better balances bias and variance in the learning updates, improving convergence and stability. They test their approach on advanced language models and show it works better than previous methods when updates are delayed.
Open → 2610.01896v1

Improving reasoning models by learning from nearby expert suggestions

Better Supervision Is Nearby: Neighborhood On-Policy Self-Distillation

Abstract: On-policy self-distillation (OPSD) trains mathematical reasoning models using a privileged teacher that sees a reference solution and supervises student-sampled prefixes. Standard OPSD uses one fixed parameter setting at every state, but nearby settings may offer additional supervision. We find that local parameter perturbations reveal complementary reference-aligned corrections under the same reference context. Different experts supply these corrections at different reference positions. Their pool covers more such positions than the unperturbed privileged teacher. We introduce Neighborhood OPSD (N-OPSD) to turn these corrections into supervision at student-visited states. Offline, greedy selection builds a compact pool of frozen experts by rewarding filtered reference-token gains beyond the pool's current best at each position. The highest-peak expert need not provide the best training target. Online routing therefore separates the anchor direction from its level of support. MaxPeak selects the anchor token, and quantile selection chooses among experts whose top token matches it. The student learns from the chosen expert's full next-token distribution through the clipped forward-KL objective inherited from OPSD. We evaluate on AIME 2024, AIME 2025, and HMMT February 2025. Across three independent runs per method, Neighborhood OPSD improves the three-benchmark Average@12 over OPSD by 2.75, 1.67, and 1.94 points on Qwen3-1.7B, 4B, and 8B, respectively. Student-prefix continuations support using the pool beyond the reference trajectories used for selection. Matched ablations support filtered reference-token gains as a selection criterion. Accounting for overlap within the pool and routing by state further improve student accuracy. Inference uses only the distilled student.

Wed 30 SeptComputation and LanguageMachine Learning
The gist
Training AI models that solve math problems can be helped by learning not just from one expert solution, but from a group of slightly different expert views nearby. The authors show that these nearby experts provide useful corrections at different steps of solving a problem, giving better guidance to the model. Their method, called Neighborhood On-Policy Self-Distillation, selects the best experts dynamically and improves model accuracy on math benchmarks compared to previous approaches. This approach learns from the full advice of the selected expert, not just the top suggestion, helping models generalize better.
Open → 2609.39687v1

Row normalization affects training speed of large language models

The Row Normalization Puzzle in Muon

Abstract: This paper examines how row-wise renormalization affects Muon, focusing on the gap between NorMuon's worst-case guarantees and its practical performance (Li et al.). Despite its growing adoption and promising performance in large language model (LLM) pretraining, NorMuon's worst-case guarantees remain poorly understood. One fundamental question is: Does row normalization yield provable convergence gains, potentially through its interaction with approximate polar computation and exponential moving-average momentum? Our results show that row normalization introduces a dimension-dependent factor in the worst-case iteration complexity under the operator-norm geometry, which persists even with exact polar computation and any fixed momentum parameters. Indeed, we establish an algorithm-dependent lower bound and a matching upper bound in deterministic settings, and extend our upper bound analysis to stochastic settings. Both upper-bound analyses allow approximate polar computation. Experiments show that NorMuon is slower than Muon on synthetic problems inspired by our worst-case construction, yet outperforms Muon in LLM pretraining. These findings sharpen the puzzle of why row normalization helps in practice and complement the recent findings of Dewulf et al.

Wed 30 SeptMachine Learning
The gist
This paper looks into how a technique called row normalization influences the training of large language models, like the ones used in AI. The authors found that row normalization can slow down the process in some worst-case scenarios but still helps in practice during real training. They proved mathematically why this slowdown happens and showed through experiments that despite this, the method works better on actual language tasks. This raises questions about why it helps in real-world cases even though theory suggests it should not always be faster.
Open → 2609.39114v1

Smaller frozen models improve training of larger code and math ai

Smaller Models, Better Rejects: Preference Distillation Scaling

Abstract: Preference distillation typically treats a teacher response as preferred and the student's own response as rejected. This assumes that self-generated failures are the most informative negatives and that rejects must come from a model at least as large as the student, making generation costly at scale. We find neither assumption holds: across students from 7B to 72B, smaller frozen models generate rejects with less inference compute yet train stronger students than self-generated rejects, before and after sequence-level knowledge distillation, on code generation and mathematical reasoning. To explain this result, we derive a finite-horizon utility bound for Direct Preference Optimization in a linearized feature model. The bound characterizes favorable reject distributions and motivates three interventions. First, mixing rejects from smaller and student-scale models improves performance as the smaller model's share increases. Second, reassigning rejects to other prompts and shuffling their code tokens still outperform length-matched gibberish, showing that task structure contributes to reject utility. Third, selecting candidates with lower likelihood under the reference policy improves net transfer when higher-likelihood candidates provide less useful contrast. Lower-likelihood selections outperform higher-likelihood ones for every source. These results suggest that effective rejects preserve task structure while limiting coupling to the reference policy, and that smaller frozen models can provide them at low cost.

Wed 30 SeptMachine Learning
The gist
Teaching AI models to understand preferences often requires comparing good responses against bad ones, but generating bad responses can be costly. The authors found smaller, fixed AI models can produce these negative examples more efficiently and help improve larger student models. They also discovered that mixing different sources of negatives and choosing those that differ more from the usual responses can make training better. This approach was tested on tasks like code writing and math problem solving.
Open → 2609.38987v1

Model improves math problem solving with activation-based learning

Activation-Conditioned Self-Distillation

Abstract: On-policy self-distillation uses a model as its own teacher to provide dense supervision for reasoning, often through reference-solution conditioning. Providing privileged information does not by itself ensure effective token-level supervision throughout long responses. We introduce Activation-Conditioned Self-Distillation (ACSD), which extracts a steering vector by contrasting activations of self-generated trajectories that reach verified correct answers within a generation budget with those of all remaining trajectories. A frozen copy of the base model applies this vector at each prediction position, and the student learns from its next-token distributions on student-generated prefixes. Outcome verification is used for direction construction and calibration; distillation requires neither problem-specific reference text nor teacher parameter updates. The distilled student is used alone at inference. On each of five models, ACSD achieves the highest mean accuracy over four mathematical benchmarks among the evaluated methods. On DeepSeek-R1-0528-Qwen3-8B, mean mathematical accuracy reaches 71.9\% and LiveCodeBench v6 pass@12 reaches 70.9\%, compared with 69.0\% and 66.3\% for the reference-conditioned OPSD baseline. Contrasts among correct trajectories also support distillation, and extracted directions can be reused across mathematical training datasets. On fixed student trajectories, ACSD maintains more stable late-position logit-update magnitudes than OPSD.

Tue 29 SeptMachine Learning
The gist
Solving math problems by machines can be difficult when they need to check long answers step by step. The authors present a new method where the model learns from itself by comparing how well it does on successful attempts versus unsuccessful ones. Using these comparisons, it creates guidance to improve future answers without needing extra help or fixed solutions. This approach improved math problem-solving accuracy across several tested models.
Open → 2609.38342v1

Self-privileged critic improves intermediate value estimation in large language models

Privy to the Foil: Recasting Value Estimation with a Self-Privileged Critic for RLVR

Abstract: Assigning credit to intermediate steps remains a central challenge in training Large Language Models (LLMs) on multi-step reasoning tasks with sparse terminal rewards, and actor-critic methods such as PPO address this by learning value functions to construct token-level advantages. Their effectiveness, however, hinges on reliable value estimation, a difficult task requiring the critic to both assess progress toward a correct solution and anticipate an evolving policy's future behavior; errors in either can compromise credit assignment and destabilize online training. In this paper, we revisit the standard state-only formulation of value estimation and propose $π$PPO, a self-privileged actor-critic framework. By reusing verified same-prompt rollouts as contrastive evidence, $π$PPO helps the critic assess intermediate reasoning against successful and failed attempts, while preserving standard policy optimization and the deployment interface. Experiments show that $π$PPO consistently improves value-estimation quality by a substantial margin and outperforms representative actor-critic and critic-free RLVR baselines on challenging mathematical reasoning benchmarks, while remaining effective even when paired with substantially smaller asymmetric critics.

Tue 29 SeptMachine LearningArtificial Intelligence
The gist
Training language models to reason step-by-step is hard because it’s tricky to decide which parts of the process deserve credit when only the final answer matters. The authors propose a new approach that helps the model’s critic compare current steps with past successful and unsuccessful attempts on the same question. This comparison makes the critic better at judging progress, which improves training and helps the model solve tough math problems more reliably.
Open → 2609.37825v1

Fact-checking agents handle source trust more reliably with new training method

Multi-Channel Mitigation of Source-Trust Shortcuts in Fact-Checking RL Agents

Abstract: Retrieval-augmented fact-checkers often receive a reliability label, such as HIGH or LOW trust, for each evidence source. These labels should adjust the model's confidence and its decision to search for more evidence, while the verdict should follow the evidence content. We introduce TrustSwap, a counterfactual test that swaps, lowers, or removes source labels while keeping every evidence text fixed, and measures its three output channels (the verdict, the confidence, and the search decision) separately. Across untrained and RL-trained models at two scales, three datasets, and two prompts, confidence and search respond to the labels as intended in 49 of 50 comparisons, yet a label change alone alters 4-23% of confident verdicts for Qwen3 models and up to 50% for an existing RL-trained fact-checker. Standard GRPO fine-tuning amplifies this shortcut at 8B in all six settings. To reduce it, we propose trust-swap augmentation (TSA), which trains GRPO on each claim with both its original and its label-swapped evidence under the same gold verdict. At 4B, TSA lowers the verdict flip rate by 7-35% (relative) in four of six settings, keeps accuracy and the intended confidence and search responses, outperforms reward-based alternatives in the main setting, and carries over to an unseen label-removal perturbation. An added consistency reward helps on the trained-on swap but not on unseen perturbations. At 8B, TSA's effect is not detectable, which makes scale the main open question.

Tue 29 SeptArtificial Intelligence
The gist
Fact-checking models often use trust labels like HIGH or LOW to decide how confident they are in information sources. The authors found that sometimes these labels can trick the model into changing its verdict unfairly. They created a test called TrustSwap to check how models react when source labels are changed without altering the evidence. Then, they developed a training method called trust-swap augmentation to reduce this problem, helping models rely on evidence content instead of just source labels. This approach works well for smaller models but needs more study at larger scales.
Open → 2609.36611v1

Language models recover meaning from corrupted inputs spontaneously

Spontaneous Context Restoration: How Language Models Recover from Corrupted Inputs

Abstract: Language models sometimes produce correct outputs even when their inputs are corrupted by deletion, replacement, or misspelling. We study the internal processes accompanying this behavior, which we call context restoration, in controlled attention-only transformers and five pretrained LLMs (1B-32B parameters) across arithmetic, reading comprehension, and multiple-choice reasoning tasks. In the attention-only transformers, restoration emerges spontaneously despite training exclusively on clean sequences, without corruption training or an explicit denoising objective. We find that context restoration follows a two-phase process: early layers localize effects associated with repair at corrupted positions, while later layers accumulate these effects at uncorrupted positions through the residual stream and ultimately concentrate them at the output position. Repair outcome is predictable from hidden states: cosine alignment with the clean state is highly predictive in attention-only models, while linear probes recover additional information in pretrained LLMs. A linear probe using only the corrupted prompt's first-block hidden state predicts failure with mean ROC-AUC 0.78. This enables failure triage under matched or even partially shifted deployment conditions and may reduce unnecessary verification or computation. Failed examples also show substantially greater nonlinearity along corruption directions. Moderate-corruption finetuning increases corruption tolerance while simultaneously reducing displacement-normalized linearization error, associating improved robustness with a more nearly linear response to corruption.

Mon 28 SeptComputation and LanguageArtificial Intelligence
The gist
Sometimes language models give the right answer even if their input has mistakes like missing words or typos. The authors studied how this happens inside the model and found a two-step process where early parts spot errors and later parts fix them. This repair ability appears naturally, even wenn the models were only trained on perfect, clean text. They also made tools that can predict when the model will fail on bad inputs, which could help avoid wasted work.
Open → 2609.35475v1

Policy optimization method improves learning from reused reinforcement data

ReSPO: Reshaped Sequence Policy Optimization for Gradient Starvation in Off-Policy Learning

Abstract: Reinforcement learning from verifiable rewards (RLVR) frequently reuses rollouts across multiple policy updates, increasing the mismatch between the current policy and the data-generating policy. We identify a sign-dependent gradient starvation problem in clipped policy optimization: clipping suppresses under-generated positive responses at the low-importance-weight tail while permitting severely over-generated negative responses to dominate the high-weight tail. To address this, we propose ReSPO (Reshaped Sequence Policy Optimization), which replaces clipping with a smooth, two-branch sequence-level kernel derived from an $α$-divergence variational objective and an exponential variance-control tilt. The positive branch preserves a nonzero gradient weight for under-generated positive responses, while the negative branch suppresses heavily over-generated negative responses. We demonstrate that ReSPO effectively learns from long positive reasoning trajectories during early training, even when accumulated policy drift relegates them to the low-importance-weight tail. On dense and MoE Qwen3 models, ReSPO accelerates early optimization, improves final training scores, and achieves higher held-out benchmark performance under a rollout reuse, validating our approach on importance-weight tail control in off-policy learning.

Mon 28 SeptMachine LearningArtificial Intelligence
The gist
Reinforcement learning methods often reuse past experiences to improve a policy, but this can cause problems when the new policy behaves differently from the data it learned from. The authors describe a specific problem where some important positive learning signals get ignored because of how gradients are clipped during training. They propose a new method called ReSPO that adjusts how the learning updates are shaped, allowing the model to better learn from rare positive examples and reduce the impact of misleading negative examples. This helps the model learn faster and perform better on tests using reused data.
Open → 2609.35433v1

Bias adjustment improves fairness in language model preference training

BA-DPO: Bias-Adjusted Direct Preference Optimization for Language Model Alignment

Abstract: Preference-based alignment methods such as Direct Preference Optimization (DPO) use pairwise preferences labeled by human annotators to fine-tune language models. However, annotators carry systematic biases toward some attributes: a name that signals a gender or an ethnicity, a persona, a language variety, a formatting convention, or length. If not properly addressed, these systematic biases can be absorbed and amplified during alignment. Existing methods address length bias or annotator disagreement, but fail to eliminate biases toward arbitrary attributes. To address this limitation, we propose Bias-Adjusted DPO (BA-DPO), a generalization of DPO that adds one bias parameter per annotator toward responses carrying a declared attribute. We prove that the objective is convex in the bias parameters and that the votes identify each annotator's bias up to a shared constant. The remaining constant is what fixes the aligned model's attribute rate: by default the rate of the reference model, or a target rate, which we use to bring a biased policy to statistical parity. On a corpus with planted biases, DPO drives the attribute from a balanced start to probability 0.96 and BA-DPO removes 81 to 95\% of that shift; on MultiPref with real annotators it removes about half of DPO's lengthening. Both hold at 0.5B with full fine-tuning and at 8B with LoRA, at no higher KL than DPO and no loss in judged quality.

Mon 28 SeptMachine LearningArtificial Intelligence
The gist
When training language models to prefer better answers, human judges can bring in their own hidden biases, like favoring longer responses or certain styles. The authors introduce a new method called Bias-Adjusted DPO that helps the model learn to ignore these biases by figuring out each judge's preferences for specific traits. This adjustment makes the model’s training fairer and produces better balanced results without losing quality. Their tests show it can significantly reduce bias effects on model outputs.
Open → 2609.35044v1

Improving multi-task AI learning by protecting task-specific knowledge

PMOPD: Task Ordering, Cycling, and Parameter-Update Subspace Protection in Multi-Teacher On-Policy Distillation

Abstract: Multi-teacher on-policy distillation (MOPD) has emerged as a popular post-training paradigm for integrating specialized capabilities in frontier language models. Existing OPD research has primarily focused on optimizing single-task distillation through objective design, distillation scope, and teacher signal construction, whereas MOPD must aggregate multiple capabilities in shared parameters and address the resulting capability seesaw, in which improving one domain suppresses capabilities acquired from another. Inspired by the distinctive update geometry of OPD, we find that parameter updates from different tasks rapidly concentrate in their respective low-dimensional subspaces during MOPD, providing a direct geometric basis for identifying and controlling cross-task interference. We therefore propose PMOPD (Projection-based Multi-Teacher On-Policy Distillation), which constructs subspace memories from the cumulative parameter displacements of different tasks and projects both gradients and optimizer updates to remove components that interfere with protected task directions. We further develop a lightweight conflict probe to characterize task interactions and guide task ordering, together with a cycling strategy that balances subspace estimation and timely task revisitation. Experiments on representative Code, Reason, and Math tasks show that PMOPD improves every evaluated capability over MOPD, raising the average score across the three tasks by 2.54 points on Qwen2.5-7B and 2.09 points on Llama-3.1-8B. These consistent gains establish geometry-aware optimization as an effective and transferable approach to balanced multi-teacher distillation.

Mon 28 SeptMachine Learning
The gist
When AI models learn several tasks at once, they can forget what they learned in earlier tasks while improving others, like a seesaw effect. The authors found that updates for different tasks happen in separate areas within the model, and they use this idea to keep tasks from interfering with each other. They created a method called PMOPD that protects each task's unique updates to keep improvements balanced. Testing showed this approach boosts skill levels across coding, reasoning, and math tasks in language models.
Open → 2609.34605v1

Fisher-informed method improves stability of feedback learning in large language models

Fisher-Informed Recalibration for Feedback-Based On-Policy Self-Distillation of LLMs

Abstract: Feedback-based on-policy self-distillation has emerged as a promising approach for enabling foundation models, more specifically Large Language Models (LLMs), to learn from their own outputs under external feedback, with a single model serving as both teacher and student. However, such methods can exhibit unstable optimization, conducive to performance collapse during training. To address this limitation, we propose FIRE (Fisher-Informed REcalibration), a dual-branch framework that recalibrates the supervision applied to correct and incorrect on-policy outputs during fine-tuning. For correct responses, FIRE replaces self-distillation with re-weighted on-policy SFT, while for incorrect ones FIRE identifies feedback components that disproportionately influence the teacher-induced update and recalibrates the feedback-conditioned target accordingly. Both branches are influenced by a token-level radius derived in part from a softmax Fisher trace. FIRE separates which direction feedback should move the model from how far the model should move in that direction, while leaving well-behaved feedback supervision unchanged. Our experiments demonstrate that FIRE provides substantially more stable self-distillation while maintaining strong downstream performance, particularly in settings where standard feedback-conditioned distillation becomes unstable.

Sun 27 SeptMachine Learning
The gist
Training large language models to learn from their own outputs can be unstable and cause performance to collapse. The authors propose a method called FIRE that helps the model better handle feedback by separating how feedback moves the model from how far it moves. This method recalibrates training signals based on whether answers are correct or incorrect, using information from the model’s internal statistics. Their experiments show that FIRE makes training more stable while keeping the model’s performance strong.
Open → 2609.34009v1

Llms learn to predict accuracy differently depending on data type

LLMs learn different forms of metacognition when trained to predict their own accuracy

Abstract: Large language models are trained to always produce an answer, regardless of whether they possess the relevant knowledge, which leads them to fabricate facts. Prior work has shown that LLMs' confidence estimates correspond poorly to their actual performance, and that fine-tuning can substantially improve them. However, what models actually learn during such training remains poorly understood. We investigate how LLMs acquire metacognitive monitoring, the ability to know what one knows, by training 10 open-weight LLMs to predict their own accuracy on factual multiple-choice questions before answering them. We find that trained confidence reflects two distinct signals. While on questions close to the training data, it tracks the model's true accuracy, in other domains, it instead tracks output consistency: the concentration of the model's answer distribution. Output consistency tracking emerges early in training and generalizes across datasets, whereas accuracy tracking develops later and remains local to the training distribution. These results suggest that calibration training may not teach models to generally detect errors they commit confidently, and they raise broader questions about the nature of metacognition in artificial systems.

Sun 27 SeptComputation and Language
The gist
Large language models normally always try to give answers, even when unsure, which can cause them to make things up. This paper studies how models learn to guess how accurate their answers are when trained to predict their own correctness. The authors found that for questions similar to training data, the models' confidence aligns with true accuracy, but for other questions, confidence mostly reflects how consistent the model's answers are rather than their correctness. This shows that models may not truly learn to recognize when they are confidently wrong, raising questions about how well they understand their own knowledge.
Open → 2609.33886v1

Group relative rl improves learning from saturated data with negative rollouts

Save Your Saturated Data: Learning Beyond Reward Saturation in Group-Based RL

Abstract: Group-relative reinforcement learning (RL) relies on reward variation among sampled responses to estimate informative relative advantages. As language models become increasingly capable, existing training data can become reward-saturated: all sampled responses to the same problem might receive equally high rewards, where the group-relative learning signals vanish and leave previously useful data obsolete. In this work, we investigate whether useful learning signals can be recovered from such saturated data. We study interventions at four levels of group-policy RL pipelines---data, rollout, reward, and advantage---and conduct extensive RL training on saturated reasoning data only. While standard GRPO on saturated data would almost always yield near-0 advantages and near-noise signals, diverse interventions successfully recycle and repurpose such data: among the proposed strategies, interventions at rollout generation are consistently most effective: nudging the policy to generate ``high-quality'', incorrect solutions introduces rollouts with poor rewards into saturated groups as negative samples, which turns out to improve GRPO by 6.4% to 9.0% across Qwen3-1.7B and 4B. Other interventions such as increasing rollout temperature or adding auxiliary rewards can also restore non-zero advantages, but yield less consistent gains. Further analyses show that effective negative rollouts require informative negative trajectories, that the method remains effective alongside unsaturated data, and that it supports iterative recycling of newly saturated examples. While increasingly stronger LLMs would render more data as saturated, our results demonstrate that don't waste your saturated data: with the right strategies they can be recycled into useful RL training signals in an increasingly data-scarce world.

Sun 27 SeptMachine LearningComputation and Language
The gist
When training AI systems that learn from groups of example answers, sometimes all the answers get the highest scores, making it hard for the AI to learn what’s better or worse. The authors looked at ways to recover useful learning signals even when all answers seem equally good. They found that making the AI generate some high-quality but incorrect answers helps by creating examples with worse scores, making the training more informative. This approach helps reuse data that would otherwise be useless, improving learning performance.
Open → 2609.33126v1

Code agent grading method improves learning from test results

Groupwise Agentic Grading and Advantage Redistribution for Code Agent RL

Abstract: Reinforcement learning (RL) for code agents often uses executable tests to provide binary rewards. With these rewards, Group Relative Policy Optimization (GRPO) assigns identical advantages to test-passing trajectories within each rollout group, overlooking differences in implementation quality and adherence to task requirements. This leaves the policy without a learning signal that favors clean, targeted implementations over those containing unnecessary or out-of-scope changes. We introduce GAGAR, a framework for quality-aware credit redistribution in code agent RL. Built on dynamic sampling that retains groups containing both passing and failing trajectories, GAGAR places all trajectories from each group in a shared workspace, where an SFT-trained agentic grader jointly inspects them and ranks the test-passing candidates. Based on this ranking, we downweight lower-ranked trajectories and proportionally rescale the advantages of all test-passing trajectories to restore their original sum. This sum-preserving redistribution retains the relative weights established by quality-based downweighting while shifting credit toward higher-quality implementations. We evaluate GAGAR at industrial scale using pre-RL SFT checkpoints of MiMo-V2.6-Flash (310B total parameters) and MiMo-V2.6-Pro (1.02T total parameters). Controlled code-only Flash experiments show improved code agent performance, reduced trajectory-length growth, and more stable training. We further apply GAGAR in large-scale mixed-task RL with both Flash and Pro. Our results support combining test-based verification with groupwise agentic grading to improve the quality and stability of code agent RL.

Sat 26 SeptComputation and LanguageMachine Learning
The gist
When teaching computer programs to write code using trial and error, we often reward them only for passing tests, giving no extra credit for cleaner or more focused solutions. The authors introduce a new method called GAGAR that compares groups of program attempts and ranks the best solutions according to quality, not just whether they pass tests. This helps the learning process prefer better code and leads to more stable and efficient improvements in code-writing agents. Their experiments with very large code-writing models show this method improves performance and training stability.
Open → 2609.32577v1

Different evidence types affect AI prompt improvement results

What Should the Reflector See? An Empirical Study of Evidence in Reflective Prompt Optimization

Abstract: Reflective prompt optimization revises instructions using examples of a model's behavior, but which evidence the reflector should receive remains unclear. We study evidence composition, visibility of examples, candidate selection and domain-knowledge policy within a single-parent Pareto-guided search. Using Qwen3.5-9B as both task model and reflector, we evaluate nine reflection strategies on five datasets. From these experiments, we find three distinct patterns. For performance improvement, Failures-only produces the largest mean test gain (+8.0 percentage points), while Balanced-mix and No-examples+Val share the best mean performance rank. For reflection effectiveness, No-examples achieves the best rank for improving sampled parents, yet yields only a 1.4-point mean test gain: local reflection success does not necessarily produce a stronger final prompt. For overfitting assessment, Failures-only and Balanced-mix share the lowest mean calibration-gap rank, while larger gaps on GPQA and IFBench show that calibration gains can overstate held-out improvement. This gap is a descriptive indicator, not a direct measure of overfitting. Together, these results show why reflection strategies should be assessed separately on final performance, parent improvement and calibration-to-test transfer.

Sat 26 SeptComputation and LanguageMachine Learning
The gist
When improving instructions given to AI models, it’s unclear which examples the model should see to do better. The authors tested different ways of showing past results to the AI and found some strategies help more than others. For instance, showing only the failures improves test scores a lot, but sometimes success nearby doesn’t mean better final results. They recommend evaluating improvement methods by looking separately at final scores, local improvements, and overfitting signs.
Open → 2609.32452v1

Masking common tokens improves language model preference fine tuning

Masking Frequent Tokens Sharpens Direct Preference Optimization

Abstract: Direct Preference Optimization (DPO) aligns language models by optimizing over sequence-level sums of token-wise implicit reward differences. However, we identify a pervasive pathology in this formulation: a disproportionately small subset of high-frequency token types dominates cumulative sequence scores while appearing symmetrically across both preferred and dispreferred responses. Specifically, under canonical Qwen tokenization on Anthropic HH-RLHF, merely 69 token types account for $55.1\%$ of all response tokens and $85.9\%$ of within-pair shared token mass, exhibiting substantially lower preference-side specificity than the remaining vocabulary. This symmetric ubiquity induces gradient entanglement and dilutes the discriminative preference signal propagated through the objective. To resolve this issue, we introduce \emph{Anisotropic DPO} (\textsf{ADPO}) and its canonical realization, \emph{Frequency-Hard DPO}. Using a fixed, label-agnostic vocabulary mask, our method zeroes the implicit reward contribution of high-frequency response tokens while assigning unit weight to informative positions, thereby suppressing gradient interference without modifying preference pairs, discarding context, or introducing learned parameters. Here, \emph{anisotropy} designates non-uniform token-level objective weighting rather than representational geometry. Extensive empirical evaluations on AlpacaEval, MT-Bench, and Arena-Hard demonstrate that Frequency-Hard DPO consistently outperforms standard DPO across Qwen-2.5-7B-Instruct and Llama-3-8B-Instruct, establishing that selectively masking shared high-frequency tokens offers an effective, zero-overhead mechanism for robust preference alignment.

Sat 26 SeptComputation and LanguageMachine Learning
The gist
Language models learn to prefer better responses by comparing answers token by token. The authors found that very frequent words like "the" or "and" happen equally in good and bad answers, which confuses the training signals. They propose a simple fix: ignoring these common words during training so the model focuses on more meaningful differences. This approach works well without adding complexity and improves model alignment on several tests.
Open → 2609.32445v1

CompassPlay rewards task proposers for guiding solver learning effectively

CompassPlay: Rewarding the Proposer for Where It Moves the Solver

Abstract: In self-play, a proposer generates verifiable tasks to train a solver. Proposer rewards often depend on the solver's success rate, but equally difficult tasks can differ in their training value. We introduce CompassPlay, a self-play method that rewards the proposer through gradient alignment. The reward favors tasks whose solver loss gradients align with those of reference tasks representing the target capabilities. It draws on a first-order approximation to learning progress and scores each eligible task without additional solver training. Our experiments show gains in performance and training efficiency. In coding self-play with Qwen2.5-Coder-7B, a small external reference set guides task generation. CompassPlay improves average accuracy over AZR's difficulty reward by 1.5 percentage points on in-domain coding and 2.7 on out-of-domain mathematics. In Lean4 theorem proving, CompassPlay matches the difficulty baseline's 150-iteration cumulative coverage with 40\% fewer GPU-hours.

Sat 26 SeptMachine Learning
The gist
Training AI to solve problems often involves generating tasks for it to practice on. However, not all tasks of the same difficulty help the AI learn equally well. The paper introduces CompassPlay, a technique that rewards the task creator based on how well the tasks improve the solver's learning direction. This method uses gradient alignment to pick tasks that better match target skills without extra training effort. Tests showed CompassPlay improved accuracy in coding and math tasks and reduced resource use in theorem proving compared to standard methods.
Open → 2609.32228v1

Large language models improve math reasoning with self-play data

Self-Play Search Distillation for Large Language Model Reasoning

Abstract: Improving reasoning abilities in Large Language Models (LLMs) requires high-quality data that exposes difficult decisions, competing alternatives, and their consequences. Data scarcity is driven by the low quality of synthetic data and the cost of human labeling. We introduce Self-Play Search Distillation (SPSD), a framework for generating superhuman synthetic data via self-play of MuZero-like networks trained on board games. SPSD uses executable environments to turn search into structured reasoning problems. At each state, the expert identifies a preferred decision, plausible alternatives, plausible opponent replies, and value estimates. By converting the self-play search records into superhuman chains-of-thought, we train LLMs with environment-grounded supervision. Although trained only on self-play search records, SPSD transfers to unseen mathematics. On Qwen3-4B-Base, it raises the mean over six mathematics benchmarks from 24.1 to 36.6 while increasing the held-out-game win rate from 15% to 45%. SPSD offers an annotation-efficient way to create high-quality synthetic data for improving LLM performance in reasoning tasks.

Fri 25 SeptArtificial Intelligence
The gist
Improving how language models solve math and reasoning problems needs lots of good practice examples. The authors created a way for AI systems to play games against themselves to generate high-quality reasoning steps without needing human help. They then use these examples to train language models, which helps them get better at solving new math problems. This method raises performance on math tests significantly and reduces the need for expensive human-labeled data.
Open → 2609.30936v1

Model improves following multiple rules by using counterfactual distillation

Counterfactual Constraint-Conditioned On-Policy Distillation for Multi-Constraint Instruction Following

Abstract: Multi-constraint instruction following requires a model to respond to a query under many simultaneously active constraints. Even strong instruction-tuned models still routinely violate some of them. Existing approaches either augment supervision with sequence- or token-level RL rewards from external verifiers or learned graders, or use on-policy distillation (OPD) against a single full-context teacher whose probability mass becomes diluted as more constraints become simultaneously active. We propose CC-OPD (Counterfactual Constraint-Conditioned On-Policy Distillation), which inverts the standard supervision-generation direction in distillation. Rather than enriching the teacher with information beyond what the student sees, CC-OPD ablates each constraint from the teacher's conditioning in turn, and constructs the per-constraint signal from the resulting per-token probability differentials. The resulting per-token leave-one-out log-likelihood shifts are summed, clipped, and added to the vanilla OPD reward as a token-level shaping term. All shaping terms are obtained from the frozen teacher, without an external verifier during distillation, and the reward equals vanilla OPD wherever the aggregate shift is zero. Across two Qwen model pairs and seven benchmarks, CC-OPD achieves the highest average among all evaluated student-training methods. A 1.5B student trained with CC-OPD surpasses its own 7B RL-trained teacher on the MulDimIF benchmark.

Wed 23 SeptMachine Learning
The gist
Following many rules at once is hard for AI models, which often break some rules despite training. The authors created a new training method called CC-OPD that checks how each rule impacts the model's decisions by leaving out one rule at a time and comparing outcomes. This approach helps the model learn exactly which parts of its output are influenced by each rule without needing extra evaluators. Tests showed this method helps smaller AI models perform better, even outperforming larger ones trained differently.
Open → 2609.27421v1

Policy aligned critic training improves language model reasoning accuracy

PACT: From Credit Assignment to Critic Alignment

Abstract: Reinforcement learning has become a central component of large language model (LLM) post-training, yet token-level credit lacks a generally accepted mathematical definition, leaving its relationship to commonly used training signals unclear. We formulate three regularity conditions, namely Completeness, Prefix Consistency, and Neutrality, and prove that they uniquely determine token-level credit. This characterization provides a unified basis for explaining phenomena across existing algorithms and guides the development of an improved actor-critic training procedure. Through this lens, an ideal teacher in On-Policy Distillation (OPD) acts as an implicit critic, yielding an expected policy gradient proportional to that induced by token-level credit. Response-level REINFORCE Leave-One-Out (RLOO) signals match the expected policy-gradient contribution of token-level credit despite their coarser granularity. We further establish approximate credit sparsity under bounded outcome rewards and show how intermediate critic errors in Generalized Advantage Estimation (GAE) can become comparable to the underlying credit. These motivate Policy Aligned Critic Training (PACT), which adopts an Actor-then-Critic update order to apply importance sampling correction to critic training and better align the critic with the updated policy. In agentic mathematical reasoning, PACT achieves 72.87% average accuracy across four benchmarks, outperforming GRPO and PPO by 8.80 and 13.16 percentage points, respectively. On SWE-bench Verified, PACT achieves a pass rate of 67.4%, outperforming PPO, GRPO, and SAO by 2.4, 2.0, and 3.8 percentage points, respectively.

Tue 22 SeptMachine LearningArtificial Intelligence
The gist
Reinforcement learning for large language models often struggles with assigning credit to individual tokens during training. This paper defines clear rules that uniquely determine how to assign this token-level credit and uses them to understand existing training methods better. The authors propose a new training method called Policy Aligned Critic Training (PACT) that better matches the critic to the updated policy, improving training effectiveness. Tests show that PACT leads to better accuracy in solving math reasoning problems compared to other methods.
Open → 2609.26355v1

Large language models lose cultural opinion diversity after fine-tuning

The Limits of Simulated Societies: How Post-Training and Survey Fine-Tuning Erase Cross-Cultural Variance

Abstract: Using large language models (LLMs) to simulate diverse human populations has the potential to transform many aspects of computational social science, yet many evaluations score the average response rather than the spread of opinion within real groups. Here, we develop a diagnostic framework that measures point accuracy alongside dispersion retention, the ratio of predicted to human standard deviation ($\dr$), on 10{,}000 respondent--question pairs from the World Values Survey (WVS) spanning twelve countries and six continents. We evaluate eleven zero-shot language models and five variants fine-tuned on WVS data with SFT, DPO, and GRPO. We identify a failure mode we term \textit{consensus collapse}, where alignment training compresses outputs toward one stereotype per group. Along the post-training trajectory from the Llama~3.1 70B base to the Tulu~3 checkpoints, the first stage, supervised instruction tuning, removes half of the spread with minimal accuracy gain ($\dr$ 1.22 to 0.59; accuracy $+0.9$ points), the later stages do not restore it, and a gap opens between WEIRD and non-WEIRD countries that survey fine-tuning then deepens while pursuing higher point accuracy. The most accurate model (Tulu~3 70B-DPO fine-tuned on WVS, 57.9\%) keeps half the human spread overall ($\dr = 0.50$) and 11\% of it for Nigeria, against 0.70--0.87 for WEIRD countries. Raising the sampling temperature to 1.0 leaves the Wasserstein-1 distance ($\wone$) to human distributions unchanged for both fine-tuned DPO models, and GRPO on Qwen~3.5 9B does not restore the spread under either an accuracy reward or a distribution-shaped reward. Mixing the aligned model with an unaligned prior raises $\dr$ from 0.51 to 0.62 on a held-out split but leaves Nigeria at 0.36. Point accuracy alone therefore misjudges these simulators, and current post-training trades diversity for consensus.

Tue 22 SeptArtificial Intelligence
The gist
Simulating different human opinions around the world with AI models is tricky. The authors show large language models lose much of the variety in responses from different cultures after being fine-tuned for accuracy. This means the models often give a single stereotype answer instead of reflecting real-world differences. They find that adjusting how the models learn and combining them with less-tuned versions can help, but only to a limited extent. So, measuring only accuracy can hide important losses in cultural diversity in these AI simulations.
Open → 2609.25760v1

FlashBoB cuts memory needs for exact gradients in long softmax attention

FlashBoB: I/O-Efficient Exact Backward-over-Backward for Softmax Attention

Abstract: Transformer models built on the attention mechanism have become a central building block in modern deep learning, yet softmax attention remains a major bottleneck for long-context workloads. While FlashAttention makes the forward and first backward passes I/O-efficient, it does not support backward-over-backward (BoB), which enables exact differentiation through the backward pass for applications such as second-order optimization, test-time training, gradient-based memory, and meta-learning. Existing BoB implementations either materialize large intermediate tensors or exhaust GPU memory at long sequence lengths. We present FlashBoB, an exact, I/O-efficient algorithm for BoB in softmax attention that keeps computation within on-chip tiles and avoids all $N \times N$ intermediate tensors, where $N$ is the sequence length. The key insight is a hierarchical affine structure in the softmax double backward: two row-wise scalars determine all outputs through affine transformations. This yields a two-pass schedule with bounded on-chip static random-access memory (SRAM) usage and minimal off-chip high-bandwidth memory (HBM) traffic. FlashBoB achieves $Θ(N^2 d^2/M)$ HBM traffic ($d$ is the head dimension and $M$ is the memory size) and, within the standard FlashAttention-style score-recomputation model, matches the inherited large-cache lower bound for exact forward attention. Empirically, it scales exact attention BoB to $N=262\text{K}$ on a single A100 80GB GPU, where prior PyTorch exact baselines fail by $N=16\text{K}$, and is up to $6.3\times$ faster than FlashBack. These results make exact second-order attention practical at long-context sequence lengths where prior implementations cannot run efficiently.

Mon 21 SeptMachine LearningArtificial Intelligence
The gist
Working with transformer models that process very long sequences is slow and uses a lot of memory due to the attention mechanism. The authors developed FlashBoB, a method that computes exact gradients during training more efficiently by avoiding large memory use and heavy data movement. This lets models handle much longer sequences on a single GPU than before, speeding up certain advanced training techniques that require precise gradient calculations. FlashBoB achieves this by cleverly reusing small internal computations and reducing the need to store big intermediate data.
Open → 2609.24089v1

Gradient imbalance impacts vision language model fine tuning differently across domains

Reassessing Global Gradient-Norm Imbalance in BLIP Fine-Tuning Across Physical Domains

Abstract: Imbalanced gradient magnitudes between the visual and language pathways of a vision-language model are often treated as a defect to be corrected. We test that premise for one family of correction, deliberately excluding adaptive, signal-driven schemes (e.g. BalGrad, OGM, PMR, CGGM), which are a mechanistically distinct class outside this study's scope. Measuring the language-to-visual gradient-norm ratio, reported in parameter-normalised form, across nine fine-tuning conditions, three seeds, and three captioning datasets spanning distinct physical domain shifts -- underwater, aerial, radiological -- we find imbalance magnitude varies markedly across domains with no predictable ordering. A plain learning-rate reduction cuts imbalance substantially and lands within a few BLEU points of the best method on every dataset. Staged freezing reduces the ratio on every domain yet never ranks first; a schedule-only control isolates freezing as the cause on one dataset but not the other two. Forcing the two gradient groups to equal magnitude drives per-parameter imbalance close to zero on every domain, yet is both the best result in the study and the worst placement among full fine-tuning methods, on different datasets, with identical settings. Reductions in gradient-norm ratio do not consistently predict captioning performance across domains, and how a given level of balance is reached matters as much as the level itself. As a secondary finding, a commonly reused LoRA configuration applied to BLIP silently adapts zero visual parameters; correcting it improves BLEU-4 on all three datasets.

Sun 20 SeptComputer Vision and Pattern Recognition
The gist
Training models that connect images and text can involve uneven updates between the visual and language parts, which people often try to fix. The authors studied whether balancing the sizes of these updates always improves performance across different image types like underwater or aerial photos. They found that how much imbalance there is and whether fixing it helps changes depending on the type of images used. Simple steps like lowering learning rates or gradually freezing parts affect imbalance differently, and forcing exact balance does not always result in better captions. They also found a common setup for fine-tuning was accidentally leaving some visual parts unchanged, and correcting this helped performance on all tested datasets.
Open → 2609.23655v1

Self-distillation methods separate learning correctness from response style

On Repulsive and Attractive Teachers: Separating Correctness from Behavior in Self-Distillation

Abstract: On-policy self-distillation provides dense, token-level supervision by conditioning a model on privileged information and distilling the resulting teacher distribution back into the model. However, privileged information can change not only what the teacher knows, but also how it behaves, entangling correctness-relevant learning signals with unintended behavioral shifts. We study this effect in reasoning tasks by contrasting attractive self-distillation, which moves the model toward a privileged teacher, with repulsive self-distillation, which moves it away from a privileged teacher. We find that both objectives can induce strong and opposing behavioral shifts: attraction suppresses exploratory reasoning and promotes shorter, more confident responses, whereas repulsion increases response length, can trigger unintended switches into a model's latent thinking mode, and ultimately becomes unstable. Motivated by these observations, we study contrastive self-distillation, which combines attraction toward a correct-solution-conditioned teacher with repulsion from an incorrect-solution-conditioned teacher. In contrast to prior work that combines such distillation signals with a GRPO objective, we isolate the self-distillation objective and study its behavior on its own. We find that the shared behavioral shifts of the two teachers largely cancel, leaving a token-level signal that more directly reflects correctness. Across non-thinking, instruct-only, and already-thinking models, this contrastive objective improves reasoning performance while maintaining stable response lengths.

Fri 18 SeptMachine LearningArtificial Intelligence
The gist
When teaching AI models how to reason better, giving them extra information can change not just what they know but how they act. The authors studied ways to encourage AI models to either imitate or avoid a teacher model's behavior, finding that pushing a model to mimic (attraction) or avoid (repulsion) leads to very different changes in answer style and stability. They then combined these approaches in a new method that encourages learning from correct answers while moving away from wrong ones, helping the model focus more on correctness and less on style. This new method improved reasoning performance and kept answers stable in length.
Open → 2609.21561v1

Paraphrase method reveals and fixes language model factual errors

Hallucination-R1: Robustness-Oriented Paraphrase Generation for Factual Consistency

Abstract: Factual hallucination is commonly defined by incorrect factual outputs. We study a paraphrase-induced hallucination setting, where a model answers a factual question correctly in its original form but generates an incorrect answer under a semantically equivalent paraphrase. Such inconsistencies expose latent factual instability under semantic invariance. However, general-purpose paraphrases are often insufficient as robustness-oriented supervision: near-copy paraphrases provide weak signals, while overly diverse paraphrases may break semantic equivalence. In this paper, we propose HALLUCINATION-R1, a robustness-oriented paraphrase generation framework that learns to produce semantically faithful yet robustness-challenging paraphrases for factual consistency. Through two-stage optimization, it first stabilizes meaning-preserving and diverse paraphrasing, then rewards paraphrases that reveal factual consistency degradation in downstream QA models. Experiments on SimpleQuestions, PopQA, and TruthfulQA show that HALLUCINATION-R1 achieves a strong consistency--diversity trade-off and exposes robustness failures across multiple model families and datasets. Further analyses indicate that these failures are not reducible to surface-level artifacts or semantic drift, but reveal non-trivial factual instability under meaning-preserving variation. A lightweight fine-tuning study also shows that HALLUCINATION-R1-generated data improves robust accuracy under paraphrase variations, suggesting its utility for robustness-oriented training. Our code and models are publicly available at https://github.com/yuwenhan07/Hallucination-R1.

Fri 18 SeptComputation and LanguageArtificial Intelligence
The gist
Sometimes AI systems give wrong answers when a question is asked in different but equivalent ways, exposing a problem with how facts are understood. The authors created a method called HALLUCINATION-R1 that generates paraphrases that are still faithful to the original meaning but tricky enough to show where AI models make factual mistakes. This method helps reveal hidden errors in question-answering models and can produce training data that makes those models better at staying factually consistent across different question forms.
Open → 2609.21227v1

Taxonomy clarifies robot responses to incomplete human instructions indoors

From Wizard-of-Oz Human-Robot Dialogue Collection to a Taxonomy of Robot Response Decisions: A Retrospective Analysis of Assistive Pilot Interactions

Abstract: Robots that follow natural-language instructions in everyday indoor environments must act on incomplete human utterances. Instructions often omit essential information, such as the identity of an out-of-view object, an intended destination, or the user's goal. Existing datasets contain little real-world situated dialogue and provide few practice-grounded criteria for deciding when a robot should act, confirm, clarify, or refuse. We retrospectively analyze a pilot Wizard-of-Oz study in which five participants performed everyday indoor tasks, including door opening, drawer opening, feeding, drinking, and cleaning, with a wheelchair-mounted mobile manipulator while the wizard responded without a formal communication policy. This preserved authentic user behavior but produced inconsistent robot-side decisions, motivating an explicit decision scheme. From 40 episodes, we derived a hierarchical taxonomy of six response modes (ANSWER, REPORT_DONE, REFUSE, CONFIRM, CLARIFY, ACT) and four ambiguity types (intent, referential, spatial, intelligibility). Two human annotators and an AI annotator applied the scheme to the pilot data. Clean-label rates were 91% and 89%, and Cohen's ranged from 0.72 to 0.95 across decision-point, mode, and ambiguity levels for both human-human and human-AI comparisons. Fine-tuning LLaVA-1.6-7B on taxonomy-derived labels for ACT and CLARIFY indicates the feasibility of training vision-language models using annotations from our taxonomy. Remaining boundary cases in decision-point identification and REPORT_DONE motivate a constrained protocol for more consistent dialogue collection.

Wed 16 SeptRobotics
The gist
Robots helping people indoors often get unclear directions because humans leave out details. The authors studied how a person pretending to be a robot responded during tasks like opening doors or feeding. From this, they created a system to categorize robot replies, like when to act, ask questions, or refuse. They tested how well humans and AI could apply these categories and explored training AI to decide by itself.
Open → 2609.19447v1

Adam optimizer loss spikes linked to momentum and loss shape

Beyond Quadratic Loss: The Stability Phase Diagram of Adam

Abstract: Loss spikes are recurrent instabilities in neural-network training and can arise from multiple mechanisms. For Adam in particular, macroscopic loss spikes have been linked to optimizer dynamics, yet how its two momentum timescales govern them remains unclear. We investigate this dependence by mapping training dynamics across the $(β_1,β_2)$ plane. Across a range of model--task settings, an approximately linear boundary, $1-β_2=C(1-β_1)$, separates spiky from non-spiky dynamics, whereas a one-dimensional quadratic loss produces approximately cubic slope. A one-dimensional superquadratic loss $L(x)\propto|x|^n$ recovers the near-linear scaling and links the boundary coefficient to the effective loss exponent $n$. We further show that confident cross-entropy losses develop a core--wall landscape comprising a narrow quadratic core followed by a steep wall, which produces effective superquadratic behavior at the scale of an optimizer update. Together, these results connect Adam loss spikes to both the mismatch between momentum timescales and finite-scale superquadratic loss geometry beyond the Hessian.

Wed 16 SeptMachine Learning
The gist
Training deep learning models sometimes experiences sudden jumps in error called loss spikes, which slow down learning. The authors studied the Adam optimizer and found that these spikes depend on how two kinds of momentum are balanced during training. They discovered a simple mathematical relationship that separates stable training from unstable, spiky training. They also showed that the shape of the error landscape beyond simple curvature explains these spikes, especially for common classification losses.
Open → 2609.18314v1

Token-level scores reveal causes and fixes for harmful fine-tuning errors

TAME: Token Attribution and Masking for Emergent misalignment

Abstract: Fine-tuning an aligned language model on narrow, flawed data can induce harmful behavior far outside the training domain, known as emergent misalignment (EM). Prior work has localized EM in model weights, activations, and training documents, but it remains unclear which training tokens carry the relevant fine-tuning signal. We introduce TAME (Token Attribution and Masking for Emergent Misalignment), a three-stage framework: token attribution scores how strongly the fine-tuning update raises each response token's likelihood, using forward passes through a released LoRA adapter; signal characterization finds patterns among high-attribution tokens; and causal validation tests them by attribution-guided loss masking. On released EM organisms and a 6,849-example medical-advice split, attribution is concentrated (the top 5% of tokens hold 32% of the mass) and, in Llama, depleted for medical vocabulary but enriched for a register of unwarranted certainty, even after controlling for token rarity. Masking high-attribution tokens during fresh fine-tuning cuts EM by 23x in Llama and 36x in Qwen, with the perplexity cost concentrated on the targeted register rather than on medical content; an equal random mask leaves EM unchanged. In Llama, the attribution pattern suggests that EM-relevant signal lies more in how confidently flawed content is expressed than in its domain vocabulary; the causal masking effect itself holds across both model families.

Tue 15 SeptMachine LearningArtificial IntelligenceComputation and Language
The gist
Fine-tuning AI language models on narrow or flawed data can cause them to behave badly in unexpected ways. The paper presents a method called TAME, which finds the specific words in the training data that cause these harmful behaviors. By marking and masking these high-impact words during fine-tuning, the researchers significantly reduce the harmful behavior without harming the model’s useful knowledge. They found that the problem mainly comes from how confidently wrong information is expressed, not just the topic words themselves.
Open → 2609.16754v1

On-policy distillation sped up by stopping early and restarting smartly

Know When to Stop, Where to Restart: Accelerating Multi-Turn Agentic On-Policy Distillation

Abstract: On-policy distillation (OPD) has become a standard approach for transferring capabilities from large teachers to compact students. Its cost, however, is dominated by autoregressive student rollouts and scales poorly in multi-turn agentic settings. Existing acceleration methods truncate or relocate the supervision signal according to fixed, offline budgets, despite substantial variation in teacher-signal reliability both within and across trajectories. Our empirical analysis on $τ^2$-bench reveals a clear structure in this variation: informative supervision is concentrated in the prefix of each turn, and, most importantly for multi-turn agentic training, the cross-turn loss of teacher endorsement is temporally locked to the student's first erroneous action rather than accumulating gradually over turns. Building on these findings, we propose STRIDE (Stop-and-Restart on-policy Distillation acceleration), which combines two complementary techniques: adaptive early stopping, which terminates a rollout once the cumulative teacher log-probability falls below an out-of-distribution threshold, and a prefix buffer, which caches high-quality prefixes and restarts generation at the weakest correct turn. Together, these mechanisms induce a data-driven curriculum that progressively extends coverage to later turns. On $τ^2$-bench retail, our method matches full-trajectory OPD and exceeds the 30B teacher at a $3.73\times$ speedup, surpasses the baseline itself at $2.34\times$, and retains a $4.51\times$ speedup under cross-domain multi-teacher training. As a supplementary generalization test beyond the agentic setting, STRIDE outperforms full OPD on AIME 2025 at a $5.10\times$ speedup and on AIME 2024 at a $3.08\times$ speedup; averaged across the two evaluations, both fixed-budget truncation baselines remain below full OPD.

Sun 13 SeptMachine LearningComputation and Language
The gist
Training smaller, faster AI agents using examples from bigger, smarter ones takes a lot of time because each step has to be carefully replayed. The authors studied how the helpful teaching signals vary over time and found that important guidance mostly appears at the start of each step, and mistakes are clear from the very first action gone wrong. Using these insights, they created a method called STRIDE that stops teaching when the guidance is unreliable and restarts from good points, making training faster without losing quality. This method speeds up training significantly across different tests while keeping performance high.
Open → 2609.14636v1

Image tokenizers influence text and image tasks in multimodal AI models

Studying Image Tokenizers as Visual Languages in Unified Multimodal Models

Abstract: Image tokenizers define the ``visual language'' of unified multimodal models, yet are commonly studied through isolated metrics or generation-/understanding-only evaluations. These evaluations do not fully capture how visual tokens behave when modeled jointly with text. We build a controlled pure-autoregressive testbed and track task-specific validation losses during multimodal continual pretraining across text, image, text-to-image (T2I), and image-to-text (I2T) prediction. We examine how these losses scale and relate to downstream performance, then use them to study multimodal learnability---how well image and text tokens are jointly modeled---and tokenizer design. We find that (1) losses should be analyzed by task, since they exhibit distinct scaling behavior and rank tokenizers differently. (2) The loss--performance relationship depends on the predicted token space: for a fixed tokenizer, T2I and I2T losses correlate with generation quality, but across tokenizers, the T2I loss--performance relationship shifts with the image-token space, whereas I2T loss, computed over a shared text vocabulary, provides a more consistent signal. I2T loss also correlates with both generation and visual understanding performance after supervised finetuning. Using losses as a lens, we show that (3) better reconstruction does not necessarily yield lower task-specific losses or stronger downstream performance, and that (4) image tokenizer choice can affect text modeling under joint optimization. As case studies, we revisit three tokenizer design axes---the discriminator, semantic supervision, and vocabulary size---to examine their effects on joint modeling and downstream performance. Together, our testbed offers a complementary perspective on image tokenizers as visual languages, highlighting their interplay with text in joint multimodal training.

Tue 8 SeptComputer Vision and Pattern RecognitionComputation and Language
The gist
Image tokenizers turn pictures into a kind of language that computers can understand alongside text. This paper studies how well different tokenizers work when images and words are learned together in an AI model. The authors found that how these tokenizers perform depends on the specific task and that better image reconstruction doesn't always mean better overall results. They also show that choosing the right image tokenizer can affect how well the model understands text when both are trained together. This helps improve how future AI handles pictures and words jointly.
Open → 2609.09143v1

Difficulty adaptive tree search improves reasoning coverage in reinforcement learning

Difficulty-Adaptive Tree-Structured Policy Optimization for Expanding Reasoning Coverage in RLVR

Abstract: Reinforcement Learning with Verifiable Rewards (RLVR) has been central to the recent success of Large Reasoning Models. However, while RLVR significantly improves single-sample accuracy, it often fails to expand the model's intrinsic reasoning coverage (pass@k) due to limited exploration during training. To address this, we optimize the structural design of train-time rollouts to enhance pass@k. Our analysis identifies three key design principles: (1) difficulty-adaptive rollout can play an important role in expanding pass@k, beyond serving as an efficiency heuristic; (2) tree-based rollout outperforms parallel sampling in discovering correct answers; and (3) sentence-entropy-guided forking overcomes the localization phenomenon of token-level branching to maximize semantic diversity. Building on these insights, we propose DATPO (Difficulty-Adaptive Sentence-entropy-guided Tree-structured Policy Optimization). DATPO integrates difficulty-adaptive tree search with a sibling-diversity advantage term, explicitly promoting semantic diversity to expand reasoning coverage during training. Experiments on mathematical reasoning benchmarks demonstrate that DATPO outperforms baselines especially in pass@k, which directly translates to superior test-time scaling performance.

Tue 8 SeptMachine LearningArtificial IntelligenceComputation and Language
The gist
Finding the best answers with AI models can be hard because they might not explore many different possibilities during training. The authors show that changing how these models explore ideas—by focusing on harder challenges and using a tree-like method—helps discover more correct answers. They also found that looking at whole sentences instead of single words during this search helps the model be more creative and cover more reasoning options. Their new approach, called DATPO, leads to better performance especially when considering multiple attempts to solve problems. This method was tested on math puzzles and did better than older methods.
Open → 2609.08650v1