Papers for

ai tool builders

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

Word editing agents struggle to infer formatting from document context

WordInferBench: Can Agents Infer How to Edit from Word Document Context?

Abstract: Document editing is among the most common knowledge-work tasks, yet existing benchmarks for evaluating autonomous agents on Word document editing provide fully specified attribute-modification instructions. They do not evaluate agents' ability to infer the appropriate attributes from document context, a capability essential in real-world editing where users expect consistency without spelling out every attribute. We introduce WordInferBench, a benchmark comprising 280 base document-editing instances, each paired with three instruction conditions: descriptive (the instruction explicitly lists every attribute change), referential (the instruction points to another object as an exemplar), and inferential (the agent must examine document context to infer what attributes to apply). To evaluate contextual attribute inference, each base instance is constructed so that the target formatting is recoverable from an in-document exemplar. The instances are derived from real Word documents collected from Common Crawl and cover 13 object types and 81 attributes. We evaluate six recent coding-agent systems and find that the best-performing evaluated system, Claude Code with Opus 4.6, achieves a 62.9% task-level pass rate on descriptive variants but only 40.7% on inferential variants, indicating that current systems complete tasks less reliably when the required formatting changes must be determined from document context.

Wed 11 NovArtificial Intelligence
The gist
Editing documents involves many small details like fonts and colors, but most AI tools follow exact instructions and don’t figure out formatting on their own. The authors created a new benchmark to test if AI agents can look at an example in a Word document and copy the style without being told every detail. They tested several recent AI coding tools and found that these tools do much better when instructions list every formatting change, but perform worse when they must guess from context. This shows current AI isn’t yet good at understanding and applying document styles just by looking.
Open → 2611.15189v1

Combining classical tools and language models improves recursive program synthesis

Symbolic Synthesis with Token-Level LLM Guidance

Abstract: We propose a synergistic combination of classical program synthesis techniques and contemporary large language models (LLMs) to solve the problem of recursive program synthesis from first-order logic specifications. Our approach leverages the strengths of both paradigms: we use LLMs as a source of statistical bias to guide the search for a correct program. At the same time we use techniques from classical synthesis to short-circuit on semantically incorrect prefixes and to avoid sampling semantically equivalent responses. The statistical bias is derived from the next-token distribution of an LLM, which is highly context-dependent. We operate with the understanding that (1) there is some class of synthesis instances that can be discharged by small models and symbolic tools, and (2) every other instance can be decomposed into these instances. Therefore, our intention is to expand the reach of small, open-weight LLMs and symbolic synthesis techniques, not to compete with proprietary frontier models on every possible instance. These are different tools for different jobs. In our experimental evaluation, we find that our method solves more benchmarks than either small LLMs or classical synthesis techniques alone. Our method solves 30/32 benchmarks using only a Qwen2.5 model with 32 billion parameters. Qwen2.5 solves 0 benchmarks on its own, and only solves a few additional benchmarks when combined with traditional constrained decoding techniques.

Sun 8 NovProgramming Languages
The gist
The paper tackles the problem of writing recursive programs from logical rules. The authors combine traditional program synthesis methods with smaller large language models (LLMs) by using the LLM's next-token predictions to guide the search for correct programs. Their method filters out incorrect and duplicate program parts early using classical techniques, improving efficiency and success rates. Experiments show this combined approach solves almost all tested synthesis problems while the LLM or classical synthesis alone solve far fewer.
Open → 2611.11057v1

Execution trace debugging improves fixing of assertion failures in ai code

On the Limits of Execution Traces for Repairing Assertion Failures in LLM-Generated Code

Abstract: Iterative repair returns a failed program's error to a large language model (LLM) and asks it to fix it. After iterative repair, the remaining failures are mostly assertion failures: with four Qwen2.5 models, they make up 79% of residual failures on BigCodeBench, all on HumanEval and 93% on LiveCodeBench. We argue, through an information view of the feedback, that an assertion failure says little about where the fault lies. We propose Breakpoint Debugging (BPD), which returns the failed program annotated inline with its variables' runtime values. We take the tasks that iterative repair failed to fix with five repair attempts and compare two ways of spending five further generations on them: BPD, and resampling. On BigCodeBench, BPD repairs 26.7% of these tasks against 22.4%, and its advantage grows with model size. On HumanEval, BPD outperforms resampling but only slightly, and on LiveCodeBench, fresh sampling repairs more (6.5% against 4.9%). Furthermore, BPD reduces unchanged-code resubmissions from 28% to 8% on BigCodeBench. Using only the first failing test's trace, it outperforms tracing all failing tests while using fewer tokens than both all-trace BPD and plain iterative repair. On LiveCodeBench, BPD gives no gain even on the public tests, so a visible failing input is not enough for a trace to help. These results match the information view: a trace helps only when the test exposes a failing input, and the fault is an implementation slip; otherwise, a repair loop should resample rather than trace.

Fri 6 NovSoftware EngineeringArtificial Intelligence
The gist
Fixing errors in code generated by large language models (LLMs) is hard, especially when the errors are about failing checks inside the program (assertions). The authors found that simply having the error message doesn’t give enough clues to fix the code. They proposed adding detailed run-time information about the program’s variables at the point of failure, called Breakpoint Debugging, which helps the model repair the code better in many cases. However, this approach isn’t always better than just trying new fixes from scratch, especially when the failing tests don’t reveal enough about the problem.
Open → 2611.09212v1

Ai coding agents keep rewriting code long after improvements stop

Your Wish Is My Commit: Over-Compliant Refactoring in AI Coding Agents

Abstract: Large language models are known to be sycophantic, over-complying with users even when doing so is unproductive. When these models act as coding agents, this can be costly: an agent that keeps editing after no edit improves the code writes churn, technical debt, and review burden into the codebase. Yet the consequences of sycophancy in software engineering remain largely understudied. To understand them, we run a controlled study of iterative refactoring, an open-ended task with no stopping criterion, where any edit an agent makes once the code is already clean reflects compliance rather than improvement. Across algorithmic (n=50) and object-oriented (n=10) suites, we repeatedly ask two widely used agents, Codex and Claude Code, to refactor further across five models. Over 3,000 turns, we define six behavioral signals of over-compliance. We hypothesized that a capable agent would stop once it recognized an optimized codebase; none did. Every model kept editing, rewriting up to 31x the original codebase before first stopping and resuming edits even after declaring the task complete. Over-compliance is a costly, multi-turn behavior that compounds with every turn, and our six signals give developers and tool builders a way to detect and curb it.

Mon 26 OctSoftware EngineeringHuman-Computer Interaction
The gist
Large language models that help with coding often keep changing code even when it's already good enough, which can cause unnecessary work and confusion. The authors studied this behavior in two popular coding agents by asking them to repeatedly improve code with no clear stopping point. They found that none of the agents stopped editing when the code was already clean, sometimes rewriting the code many times over. The paper identifies signs of this over-compliant behavior and suggests ways developers can detect and reduce it.
Open → 2610.36346v1

Contrastive decoding improves reasoning reliability in large language models

Mitigating Query Shortcuts in Chain-of-Thought with Context-Aware Decoding

Abstract: While Chain-of-Thought (CoT) prompting improves the reasoning performance of large language models (LLMs), recent studies have shown that final answers do not always depend on the generated rationales. In particular, LLMs may rely on query-specific shortcuts rather than the information provided by their reasoning steps. In this paper, we investigate whether context-aware decoding (CAD), an existing contrastive decoding method, can mitigate such shortcut behavior when applied to CoT reasoning. We study this application as Rationale Contrastive Decoding (RCD), which contrasts the next-token distributions conditioned on the query with and without its rationale, amplifying tokens whose likelihood increases due to the rationale. We provide a theoretical interpretation of this decoding rule via a KL-regularized reward-maximization objective. Experiments across five reasoning benchmarks demonstrate that RCD improves answer accuracy over standard CoT. Furthermore, intervention analyses using ground-truth and random rationales reveal that RCD increases the behavioral dependence of final answers on the rationale content. Our findings suggest that contrastive decoding offers a simple, training-free approach to mitigating query shortcuts.

Wed 21 OctComputation and LanguageArtificial IntelligenceMachine Learning
The gist
Sometimes, large language models give the right answers without actually using the detailed reasoning steps they explain. This means they might just take shortcuts based on the question instead of truly thinking through the problem. The authors studied a new way called rationale contrastive decoding, which helps the model rely more on its reasoning steps when deciding on answers. Their tests show this method improves accuracy and makes answers depend more on the reasoning provided. This approach works without needing to train the model further.
Open → 2610.29518v1

AI helps developers do new work and learn together

"To Boldly Go Where No One Has Gone Before" What Has AI Let Developers Unlock?

Abstract: Measuring AI by how much faster developers work is like judging electricity by how much faster it spun a steam-era factory. A fuller measure asks what new work becomes possible. We surveyed 331 software developers at Microsoft about what AI lets them do that they could not do before and analyzed their answers using reflexive thematic analysis with a human-in-the-loop, multi-model AI council. Developers described doing familiar work faster (Faster, 45%), undertaking work previously beyond their reach (Further, 36.9%), and taking on new kinds of work centered on building for and directing AI agents (Afresh, 12.4%). Across these accounts, learning and scaffolding ran in both directions: developers learned new skills with AI while externalizing their tacit knowledge as the context and standards that guide AI's work. Drawing on Vygotsky's zone of proximal development, we frame developers and AI as joint learners whose zones meet at the skill-judgment nexus: AI can supply skill faster than it can reliably judge its own work, and developers must continuously bridge that gap. Whether they do sets them on one of two paths. On one, developers' judgment grows alongside AI's skill, and they codify what they learn so the loop keeps turning. On the other, they become hollow orchestrators, approving work they can no longer judge and losing the identity and meaning they once drew from it. Based on our findings, we offer a two-way scaffolding playbook to help developers, teams, and tool builders take the first path.

Fri 16 OctHuman-Computer Interaction
The gist
Measuring AI only by how much faster it helps developers misses a bigger question: what new tasks can AI help them achieve? The authors surveyed Microsoft developers who said AI lets them work faster, reach tasks they couldn’t do before, and create new AI-focused work. Developers and AI learn together: AI gets skilled quickly but can’t always judge its work well, so developers need to guide and improve the AI. This cooperation can enrich developers’ skills or make them lose confidence if they rely too much on AI without understanding it.
Open → 2610.23708v1

LLMs generate code confidently even when execution fails often

Characterizing Overconfident Failure in LLM-Based Code Generation

Abstract: Large language models (LLMs) are increasingly used for automated code generation, but generated programs can appear syntactically plausible while still failing execution-based correctness checks. Existing validation methods, such as testing and program analysis, remain essential but are often incomplete, costly, or applied only after generation. Model-derived uncertainty is therefore a natural early reliability signal. This paper studies the dilemma of overconfidence in code LLMs where incorrect programs are often generated with token-level confidence comparable to correct programs. We study this dilemma across four open-source code models and three execution-based benchmarks. Our analysis begins by investigating whether existing uncertainty metrics provide reliable proxies for execution correctness in code generation. We then characterize overconfidence at both global and local token levels, asking whether incorrect programs remain indistinguishable from correct ones under confidence and entropy summaries, including selective generation and the limits of instruction tuning. Finally, we evaluate whether common mitigation strategies reduce this failure mode. Our study yields four findings. First, existing uncertainty signals provide only partial and model-dependent evidence of execution failure. Second, overconfidence persists at both program and token levels, and uncertainty-based selection does not consistently improve accepted-set accuracy. Third, instruction tuning can increase certainty on failing generations without consistently improving correctness discrimination. Fourth, common mitigation techniques improve specific aspects of reliability but do not reliably resolve overconfident failure. Our exploratory latent analysis suggests that hidden representations may encode correctness-related signals that output confidence does not expose.

Thu 8 OctSoftware EngineeringArtificial Intelligence
The gist
Large language models used to write computer code sometimes produce programs that look right but don't actually work. The authors found that these models often give high confidence to wrong codes, making it hard to tell correct from incorrect just by looking at their confidence scores. They tested different methods to spot or reduce these overconfident errors but found these approaches only partially help. Interestingly, the authors suggest that hidden information inside the model might hold clues about correctness that current confidence measures miss.
Open → 2610.11300v1

Code models fixed faster with smarter pinpointed neuron patches

Mind the Gaps: From Failure Attribution to Closed-Form Repair of Code Language Models

Abstract: Code language models must be maintained like the software around them: when a library evolves, a model keeps writing the interface that it saw during training. Repairing the model itself lets one correction reach all downstream uses. Existing repair methods attribute a failure to neurons, select the highest-ranked ones, and apply a generic update. This pipeline assumes that the attributed neurons are the ones to patch and that a generic update fits every failure, and neither assumption has been examined. We examine both on executable API evolution tasks in Python and Rust with three code models and identify two gaps. The targeting gap separates the neurons that failure attribution targets from the neurons that can carry a patch: their top sets have a Jaccard overlap of only 0.15 to 0.20. The tailoring gap separates a generic update from a patch built for the failure: the patches that different carrier neurons need are nearly orthogonal. To address both gaps, we propose ASTRA. It targets neurons by contrastive semantics, an attribution that scores a neuron by its contribution to the logit contrast between the target token and the produced token. It then tailors the patch by solving one small linear system in closed form, which corrects all failing tokens of a sample jointly and needs neither an optimizer nor a backward pass. Contrastive semantics selects significantly better carrier neurons than gradient-based attributions in 3 of 6 settings and comparable ones in the others. On average, ASTRA reaches 66.7 percent Pass@1, against 48.4 percent for the best of AlphaEdit, STAR and low-rank adaptation, and repairs a sample in 2.7 seconds. It is the best method in all 6 settings and for every type of API change, and this advantage persists under an unseen phrasing of the test prompt. Its side effects on unrelated code are small on the large models and larger on the small one.

Sun 4 OctSoftware EngineeringComputation and LanguageMachine Learning
The gist
Code language models sometimes make mistakes when programming libraries change, because they remember old interfaces from their training. The paper finds that current methods for fixing these mistakes target the wrong parts inside the model and use one-size-fits-all fixes that don't work well. The authors introduce a new way, called ASTRA, that better identifies the neurons causing errors and applies precise fixes quickly without complex calculations. Their approach improves correction accuracy and works reliably across different programming languages and kinds of library changes.
Open → 2610.05277v1

How revising instructions improves coding agents’ rule following and accuracy

Agent Skill Evolution: How Revisions Affect Coding Agents

Abstract: Agent Skills, the SKILL.md files that tell an LLM coding agent how a project works, are revised like code, yet what a revision does to the agent is unknown. From 2,608 first/last revision pairs of 3,159 Skills, we characterize how Skills evolve and how they change together with the configuration of the agent's harness. We then focus on rule changes, revisions that add or remove a rule we can check automatically, such as "run allium check". We measure their effect on 21 models in single answers and on four agents in a sandbox, and their cost on 20 of these models and the four agents. Most revisions (55%) change a rule or procedure, and commits that revise a Skill change harness files such as CLAUDE.md more often than other commits of the same size. Across 16 open-weight models, an added rule raises compliance in a single answer by +0.41 on average. Across the four agents, the rate at which the agent takes the required action rises by +0.23 on average (+0.16 to +0.36), and for the three agents that blind judges assessed, final correctness rises by +0.10 on average (+0.06 to +0.14). The gain comes mainly from rules that name a command or path the old Skill did not mention. Real tools load a Skill's body only when the agent decides it needs it. In that setting the four agents keep about half of the action gain on average (51%), and the three open models about 38%. A revision adds 18-19% input tokens to a single answer and no detectable cost to an agent episode, while loading a Skill's body raises the tokens of an episode by 50% on average.

Sun 4 OctSoftware Engineering
The gist
The paper studies how changes to 'Agent Skills'—files that teach AI coding helpers how to handle projects—affect their performance. The researchers found that adding or changing explicit rules in these files helps AI models follow directions better and be more correct in their answers. These improvements mostly come from introducing new named commands or paths the AI didn’t know before. Although editing these skills adds some extra work for the AI, the benefits in accuracy are significant, especially when the AI uses only the parts it needs.
Open → 2610.04832v1

WitnessGym creates automated tests for validating software bugs

WitnessGym: Benchmarking Coding Agents on the Construction of Bug Witnesses

Abstract: Bug validation asks a coding agent to produce an executable witness for a reported bug. The witness combines a concrete input with a testing harness and exposes faulty behavior during execution. Such evidence makes audit findings actionable, yet benchmark evaluation is difficult when cases reuse public historical bugs and witnesses or require manual construction. We present WitnessGym, an automated framework for constructing bug-validation benchmarks through bug injection. It injects bugs into test-reached paths of real projects, rebuilds each project, and retains cases exposed by a construction-time witness. Bug specifications and execution adapters allow extension to additional bug types and languages. Bug-preserving transformations vary the surrounding structure while preserving the witness behavior. Based on real-world Java projects with test suites, WitnessGym automatically constructs 1,300 benchmark cases. The injected patches resemble historical bug patches and are difficult for the two evaluated models to distinguish in blinded comparisons. We evaluate four coding agent frameworks in six framework/model pairings across bug types, execution contexts, and transformation depths. Witness construction remains difficult even when the bug pattern is known. Our framework, benchmark cases, and evaluation scripts are available.

Tue 29 SeptSoftware EngineeringArtificial Intelligence
The gist
When software has bugs, it's important to show exactly how to trigger them to fix the problem. The authors created WitnessGym, a tool that automatically inserts bugs into real software and generates tests that prove the bugs exist. This makes it easier to evaluate how well coding AI systems can find and demonstrate bugs. They tested several AI coding models and found that producing these bug proofs remains challenging. WitnessGym and its test cases are publicly available for further research and development.
Open → 2609.36635v1

Game logic benchmark tests coding agents on detailed gameplay rules

GameLogicBench: Evaluating Coding Agents on Runtime Game Logic with Tick-Level State Assertions

Abstract: Coding agents can modify and test code across large software projects. Game development is a domain where agents must implement gameplay rules. A game can end in a valid state even after violating its rules during the run. Current game-development benchmarks replay fixed examples, score videos, or ask another model to judge the result. However, no existing benchmark checks game rules throughout execution across varied evaluator-selected scenarios while ensuring exactly reproducible verdicts. We introduce GameLogicBench, a benchmark of 72 gameplay-logic tasks in Godot projects. An automated evaluator checks each game's rules at every simulation tick. Across 403 hand-designed scenarios, seeded parameter variations produce 1,451 test cases. To ensure that the evaluator measures behavior rather than implementation choice, it must accept different correct implementations for each task while rejecting mutants, implementations with one required capability removed. The tasks span isolated mechanics, multi-system interactions, and repository-scale features. Across 20 combinations of language models and scaffolds, the best observed run solves 52.78% of tasks. Under Claude Code, all twelve models solve fewer tasks as task scope expands from isolated mechanics, through interacting systems, to repository-scale features. Agents inspect code more often and make more tool calls on repository-scale tasks than on isolated-mechanic tasks. Most unsuccessful submissions are runnable, but implement some required game behavior incorrectly. We compared versions of our benchmark evaluator built with and without validation using mutants. Without this validation, incorrect agent submissions passed. A separate analysis finds agents copying code from public repositories when network access is open. Reliable evaluation thus depends both on what the tests reject and on what external code agents can access.

Fri 18 SeptSoftware EngineeringArtificial IntelligenceComputation and Language
The gist
It can be hard to check if computer programs that write game rules are doing everything right while the game is running. The authors created a new test called GameLogicBench that looks at many parts of a game at every small step to see if it follows the rules correctly. Their test includes many different game challenges and can tell if the rules are broken at any point, even if the game ends okay. They found that current coding models still struggle with bigger, more complex game tasks. Their test also makes sure agents don’t just copy public code to pass the tests.
Open → 2609.21562v1