Papers for

ai-powered code completion teams

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

Lineage preserving sampling improves diffusion language model reasoning accuracy

Keep the Lineages Alive: Diversity-Preserving SMC for Diffusion LLM Decoding

Abstract: Inference-time sequential Monte Carlo (SMC) decoding can improve the reasoning accuracy of masked diffusion language models (diffusion LLMs) without post-training by resampling a population of partial generations according to a reward. Its effectiveness therefore depends on the resampling step, which must concentrate computation on promising candidates while retaining enough diversity to explore alternative reasoning paths. The reward can be internal, the decoder's own confidence on the committed prefix, or external, a process reward model scoring that prefix. Under either reward, multinomial resampling can disrupt this balance by duplicating high-weight particles and discarding lineages that may still reach a correct solution. The effective sample size (ESS) does not register this loss: multinomial resampling resets it to its maximum even as distinct lineages disappear. To retain such lineages, we introduce lineage-preserving SMC (LP-SMC), which combines Chopthin resampling with an input weight floor. Instead of equalizing weights, Chopthin bounds the ratio between the largest and smallest weights it returns and carries the unequal weights forward. The floor limits how far below the maximum a weight may fall before resampling so that lineages with low early weight keep a chance to recover. Across mathematical-reasoning and code-generation benchmarks on two base models, LP-SMC raises oracle coverage, the fraction of problems with at least one correct particle, in every setting, by 1.8 to 25.6 points, and raises end-to-end accuracy under a common reward-model selector in eleven of twelve settings, by up to 7.9 points on LLaDA-1.5 and 18.3 on Dream-7B. The coverage gain persists under the external reward model, which suggests that diversity-preserving resampling and the guidance signal contribute separately to inference-time search.

Mon 19 OctMachine LearningArtificial IntelligenceComputation and Language
The gist
Language models that generate text step-by-step can be improved by looking at many possibilities at once and choosing the best paths. However, common methods tend to focus too much on the most likely sequences, losing less obvious but potentially better answers. The authors present a new way to keep a diverse set of possible text paths alive during the process, which leads to better problem-solving and coding results by preserving different reasoning directions. Their method balances focusing on strong candidates while allowing weaker ones to recover, improving overall success rates.
Open → 2610.25932v1