Papers for

devops teams

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

Natural language helps improve Rust code quality from C translations

Beyond Source Code: Evaluating Natural Language Specifications for LLM-Based C-to-Rust Translation

Abstract: Large language models (LLMs) have shown promising results for C-to-Rust translation, yet existing approaches primarily rely on source code as input, leaving natural-language specifications of program behavior underexplored as an alternative or complementary representation. We hypothesize that direct code translation encourages LLMs to preserve C-specific implementation choices, producing Rust that may be functionally correct but unnecessarily verbose, reliant on unsafe constructs, and less idiomatic. We investigate this hypothesis by comparing three translation settings: code-only, specification-only, and code-and-specification. Using 300 C programs from the IBM CodeNet dataset, we evaluate GPT-5.2 and GPT-4o-mini on functional correctness, maintainability, complexity, idiomaticity, unsafe construct usage, and cost. Code-only translation achieves the highest test pass rate (98.6%) but often mirrors C's implementation patterns. In contrast, specification-only translation produces the most idiomatic and compact Rust, reducing unsafe blocks by approximately 6x and Clippy style warnings by 2.4x. However, using specifications without validation yields test pass rates as low as 86% with our best performing model. Adding an iterative test-based validation and refinement step for specifications raises this to a 95.3% test pass rate. Overall, specification-based translation consistently improves code quality relative to code-only translation, while code-and-specification offers the best balance between correctness (98.3% test pass rate) and maintainability. Our study systematically compares these translation settings, highlighting the trade-offs among specification quality, translation correctness, code quality, and computational cost.

Thu 12 NovSoftware Engineering
The gist
Translating C programs into Rust code usually relies on the original C code itself. The authors found that using natural language descriptions of what the program should do can help large language models create Rust code that is cleaner, safer, and more in line with Rust coding practices. However, relying only on descriptions can lower correctness, so combining code and specifications and using testing steps leads to the best balance of correct and high-quality Rust code. This study compares ways to translate from C to Rust, showing trade-offs between accuracy, code quality, and cost.
Open → 2611.17581v1

Runtime controls coverage measured for llm agent actions

guardscope: Measuring Which Actions Runtime Controls Check in LLM Agent Systems

Abstract: Large-language model (LLM) agents execute actions on external systems through tool calls, and enterprises deploy runtime controls, such as agent gateways, to check this behavior in real time. However, a control can only evaluate the actions that cross the point where it is placed, and an agent can act through more paths than the tool-call channel; no existing work measures this coverage against kernel-level ground truth. In this paper, we present guardscope, a framework that makes this coverage measurable. guardscope works in two stages: a static stage that predicts which of an agent's 15 action paths a control can check from its source, and an empirical stage that runs a fixed workload on a deployment that realizes one of eight topology designs and classifies every executed action as checked, bypassed, or prevented by reconciling the control's decision records against kernel-level ground truth. guardscope makes a control's coverage claim verifiable: a runtime control developer can publish which paths its product inspects under a kernel-verified protocol. We conducted experiments with two agent harnesses and real runtime controls, including Microsoft's agent governance toolkit and agentgateway. guardscope is available at https://github.com/VulcanLab/guardscope.

Wed 11 NovArtificial Intelligence
The gist
LLM agents use multiple ways to interact with outside systems, but security checks can only see some of these actions. The authors created guardscope, a tool that measures which agent actions a security control actually monitors by comparing reported checks with a low-level record of all actions. This helps developers know exactly what their controls watch for and what might slip through. The authors also tested guardscope with real tools and multiple agent setups.
Open → 2611.15223v1

Agents follow some coding instructions but often ignore others

Do They Do as They Are Told? Investigating Coding Agents' Adherence to Explicit Instructions in Context Files

Abstract: Developers increasingly rely on agent context files (ACFs), persistent, project-scoped artifacts such as AGENTS.md, to steer the behavior of autonomous coding agents without renegotiating expectations at every interaction. Yet this governance mechanism rests on an untested assumption: that instructions written into an ACF are actually carried out, rather than ignored or loosely followed. Existing studies describe what ACFs contain, but not whether individual instructions are followed, leaving open which guidance can be trusted. We address this gap through an empirical study of real-world ACF snapshots and their linked agent-generated commits: characterizing the explicit, actionable instructions ACFs encode, operationalizing them into executable measurement scripts, and using these scripts to assess instruction-level compliance. We find that most observable instructions can be operationalized and that agents follow ACF guidance more often than not, but adherence splits sharply between instructions followed consistently and those never respected, with differences across instruction categories and coding agents. This split shows where ACF guidance works and where it does not, pointing to instruction types that may need rewriting and to agents that may need enforcement beyond the ACF. These findings lay the groundwork for systematic approaches to specifying, verifying, and enforcing project-specific instructions in autonomous software development.

Tue 10 NovSoftware Engineering
The gist
Coding agents are given sets of instructions in special files meant to guide how they write code for projects. The authors studied whether these agents actually follow the instructions or just ignore them. They found that while agents do follow many instructions, there is a clear split: some commands are consistently obeyed, and others are often ignored. This shows which instructions work well to guide the agents and which need changes or extra enforcement.
Open → 2611.13709v1

Deliberate pacing balances human oversight and autonomous software development

Deliberate Pacing for Human Oversight in Agentic Software Development

Abstract: Agentic AI increasingly enables software development activities to proceed without continuous human involvement. While this autonomy may accelerate development, it also raises questions about when developers can meaningfully oversee software development done by agents. We propose deliberate pacing: intentionally adjusting when and how autonomous development proceeds to preserve opportunities for meaningful oversight. Drawing on an exploratory multivocal literature review of 20 sources, we identify three factors motivating this vision: changing relative costs of human involvement, the timing of oversight opportunities, and interdependencies between pacing decisions across the software development life cycle. We argue that neither uninterrupted autonomy nor frequent human intervention is universal. Instead, pacing should balance opportunities to understand and influence consequential decisions against costs in human attention, development time, and rework. We outline a research agenda to investigate when deliberate pacing is beneficial, how intervention timing affects oversight, and how pacing decisions interact across development activities.

Fri 6 NovSoftware Engineering
The gist
Software development is increasingly done by AI agents working on their own. This raises the problem of how and when humans can effectively check the work and make decisions. The authors propose a strategy called deliberate pacing, where the speed and timing of autonomous development are controlled intentionally. This approach aims to keep chances for human oversight meaningful without slowing down development too much or causing extra work. They identify factors like timing of review, cost of human attention, and dependencies across development stages that affect how pacing should be done.
Open → 2611.09550v1

Routine operations can silently break LLM agent harness architecture

When Routine Operations Break Architectural Pattern Relationships in LLM Agent Harnesses

Abstract: LLM agent harnesses distribute tool mediation, state management and safety guardrails across native services, independently written plugins and configuration. Routine operations can change the runtime order or retained state that architectural pattern relationships require, without any code change. We study five relationships associated with the Guardrail Enforcement Point, Autonomy Controller and Blackboard patterns across five released harnesses, analysing a main sample of fourteen independently written plugins. Using PatternOps, our assessment procedure, we trace requirements from pattern descriptions, provider documentation and deployment profiles to the providers that realise them and the records that decide them. We then compare alternative allocations under supported operations. Such operations broke relationships silently: after a plugin was reapplied or an allow rule saved, policy stopped evaluating edits while every plugin stayed active, and a prohibited edit executed. Which alternatives qualified depended on the requirement. Allocations that kept the required order, or kept or rebuilt the needed state, held on their tested paths and operation sequences, while others prevented prohibited effects but missed edits rejected elsewhere. In these harnesses, a pattern relationship is a property of the running system and its history, so architects should evaluate it against the decision paths, state lifetimes and operation sequences their deployments use.

Fri 6 NovSoftware EngineeringArtificial Intelligence
The gist
Sometimes, regular updates or changes in large language model (LLM) agent setups can unexpectedly disrupt important connections between different tools and protections without changing any code. The authors studied how certain safety and control patterns were affected across multiple setups and found that operations like reapplying plugins or saving rules could stop safety checks from working properly, letting forbidden actions happen unnoticed. Their work highlights that these relationships depend on the system’s current state and history, so system designers should carefully check how their operation sequences affect these crucial connections.
Open → 2611.09088v1

Repository context shapes security of code from large language models

A.S.E 2.0: Understanding How Repository Context Impacts the Security of LLM-Generated Code

Abstract: Large language model (LLM) coding assistants and agents construct a repository context before generating code. For security-sensitive code, this context decides whether the model sees the project's safeguards (e.g., authorization checks) or insecure code to imitate. However, context construction is designed almost exclusively for functional correctness, and secure-code benchmarks treat the context as fixed, so how repository context affects the security of generated code remains unclear. We construct CASE-Bench, a benchmark of 145 repository-level tasks from real CVEs in 51 projects. Each task runs in Docker, has functional tests and a proof-of-concept exploit validated in both directions, and is annotated with the repository code that enforces or undermines its security. Using CASE-Bench, we compare 13 context construction methods and six controls with six LLMs, edit single context elements, and analyze coding-agent trajectories. We find that (1) functionality-oriented context raises the functional pass rate by up to 8.2 percentage points (pp) but the secure pass rate by at most 3.7 pp, (2) withholding a safeguard lowers the secure pass rate by up to 8.1 pp, while one inserted insecure exemplar raises correct-but-exploitable code by 6.3 pp, and (3) coding agents observe the relevant safeguard in only 27-36% of trajectories. These findings inform security-aware context construction and secure-code evaluation.

Wed 4 NovCryptography and SecurityArtificial IntelligenceSoftware Engineering
The gist
When AI tools generate code for software projects, the surrounding code they see influences how secure the new code is. The authors found that AI helpers doing well on making code work doesn't always mean the code is safe from hacks. They created a benchmark of real security problems to test how different ways of showing context to AI impact security. Their findings show that missing security checks in the context or showing insecure examples can make AI write vulnerable code more often.
Open → 2611.06294v1

Agent skills change over time with focus on enrichment and maintenance

How Agent Skills Evolve: A Large-Scale Empirical Study of Skills on GitHub

Abstract: Large language model-based agents increasingly rely on external skills to reuse task-specific knowledge without retraining the model. As skills become common for extending agent capabilities, understanding creation and evolution is critical for reliable agent systems. However, existing studies mainly examine agent skills from a static perspective, leaving their evolution in real-world repositories largely unexplored. This limits our understanding of the patterns and motivations behind skill changes. To fill this gap, we present a large-scale empirical study of agent skills, capturing 84,200 skills and 750,898 commits from GitHub. We manually construct taxonomies of skill components, change patterns, and evolution motivations, and build an automated annotation pipeline to label changes at scale. Based on the taxonomy, we conduct quantitative analyses of skill composition and post-creation evolution frequency, together with qualitative assessments of modified components, change patterns, and motivations. Our findings reveal five implications: (i) reliability receives less attention in skill composition and change, motivating execution-contract skill design that incorporates verification and recovery mechanisms; (ii) evolution changes concentrate in project-coupled skills, calling for automated just-in-time skill updating mechanisms; (iii) change patterns are dominated by incremental enrichment and refinement, motivating pattern-guided skill enhancers that diagnose and improve incomplete skill components; (iv) redundancy is a practical maintenance issue, highlighting the need for context-efficient skill representation; (v) security-motivated changes show limited risk reduction, suggesting skill regression testing methods that objectively verify skill functionality and security. Beyond these insights, our dataset provides a basis for future studies on automated skill techniques.

Tue 3 NovSoftware Engineering
The gist
Many computer programs called agents use little pieces of code called skills to perform tasks. The authors studied how these skills grow and change by looking at a large amount of code from GitHub. They found that most skill changes improve or add new features, but not many focus on making skills more reliable or secure. This study helps show what parts of skills need more attention to make agents work better and safer.
Open → 2611.04456v1

Web fraud attacks against coding agents exploit earned trust

The Long Con on the Web: Earn-then-Exploit Fraud Attacks against Coding Agents

Abstract: Coding agents increasingly operate with web access and high privileges, creating a new attack surface where web fraud can directly target autonomous agents. Existing studies mainly focus on malicious links, prompt injection, and interface deception, while content-level web fraud against agents and the amplification of attacks through agent trust remain underexplored. In this paper, we present EtE (Earn-then-Exploit), a web fraud attack framework against coding agents, in which a malicious website first provides multiple rounds of genuinely verifiable assistance to establish trust and then introduces task-related instructions that cross intended operational boundaries. To systematically evaluate its effectiveness, we construct 56 organizational environments (8 task contexts x 7 attack consequences) and evaluate three representative coding agents. EtE achieves an average scene-level attack success rate of 71.4%, including 85.7% on Codex-GPT-5.5, 67.9% on ZCode-GLM-5.3-Flash, and 60.7% on QwenCode-Qwen3.7-Plus, significantly outperforming all injection-based baselines. More importantly, our experiments reveal that coding agents exhibit trust-like behavioral reliance on previously helpful Web sources, and that such accumulated credibility significantly increases their susceptibility to subsequent malicious instructions. Based on this finding, we propose action-authorization separation (AAS), which prevents web content from directly authorizing privileged actions and reduces the attack success rate to 0% while preserving 100% task completion, outperforming all evaluated defense baselines.

Sat 31 OctCryptography and Security
The gist
Coding agents that access the web and have high privileges can be tricked by malicious websites. This paper shows how bad websites first help these agents to build trust, and then exploit that trust to give harmful instructions. The authors tested their approach across many scenarios and found it works much better than earlier attacks. They also found coding agents tend to trust websites that previously helped them. To stop this, the authors propose a defense that separates trusted actions from privileged ones, making attacks ineffective without harming performance.
Open → 2611.00725v1

Software factories coordinate coding agents with minimal orchestration

Instant Software Factories

Abstract: Coding agents can quickly build prototypes, but a long tail of fixes, optimizations, tests, and releases separates a prototype from a product. A software factory handles this ongoing work through a repeatable process. We ask whether this work needs dedicated orchestration and how such factories should be judged. Given a tested repository and a coding agent, an instant software factory uses only two coordination inputs, a swarm of cron-job agents with standing objectives and a shared communication board, such as GitHub, where agents coordinate through the traces they leave. We developed a bootstrap prompt designed to set up both inputs from one round of human input, with governance as a separable layer. We distilled it from two manually configured industrial factories, observed for 15 and 17 days, and have not yet evaluated it. In the GitHub-coordinated CLI factory, daily merges rose 15x during agent rollout, an observed association. Its agents also filed 95% of new issues and authored 40 of 58 PRs that changed CI workflows, and at least 5.6% of issues marked as fixed were later found to have incorrect fixes. Unlike an agent scored on a fixed benchmark, a factory thus changes its own workload, evaluator, and measured progress. We formulate factory development as multi-objective optimization under this endogenous feedback. Over five months, we plan to evolve both factories, test the bootstrap on new repositories, and test five falsifiable hypotheses.

Mon 26 OctSoftware Engineering
The gist
Building software prototypes quickly is easy, but turning them into full products needs ongoing fixes, tests, and updates. This paper looks at 'instant software factories'—systems where many simple coding agents work together using shared communication tools like GitHub, with little direct coordination. The authors created a setup to start these factories from just one human input, based on lessons from two real industrial factories. They observed increased activity and found the process changes its own workload and evaluation, suggesting software development here is a complex evolving system.
Open → 2610.36254v1

Skill dreaming builds evolving code skills from pull request history

Skill Dreaming: Evolving Repository Skills through Pull Request Streams

Abstract: Large language model (LLM) agents increasingly rely on reusable skills for complex software engineering tasks, making skill libraries essential for repository development and maintenance. However, repository-specific skills must evolve with changing code structures, implementation conventions, and maintenance requirements. Pull requests (PRs) provide a natural source of such knowledge by recording how repositories evolve through implementation and review. Meanwhile, new PRs must follow repository-specific implementation patterns, modify related components consistently, and ensure that existing tests continue to pass while adding coverage for new functionality. This creates a mutually reinforcing cycle: PRs provide knowledge for updating skills, while these skills improve future PRs, maintain repositories, and support further skill evolution. To enable this co-evolution of PRs, repositories, and skills, we propose Skill Dreaming, a framework that learns from PR streams to guide continuous repository development and maintenance. Given a repository and its PR history, Skill Dreaming builds a repository-specific skill library by extracting executable knowledge from validated PRs and continuously refines the skills as new PR information becomes available. These skills guide coding agents not only in implementing requested features but also in maintaining associated tests, documentation, compatibility, and repository conventions. By integrating PR analysis, skill management, code execution, and continuous integration, Skill Dreaming provides an end-to-end workflow for applying learned skills to repository changes and incorporating subsequent PR outcomes into future skill updates.

Mon 26 OctArtificial IntelligenceSoftware Engineering
The gist
Software projects change often, and tools that help write code need to keep up with those changes. The authors created Skill Dreaming, a system that learns from past code updates called pull requests to build and improve specific skills for a project. These skills help new code fit well with existing code, tests, and documentation. Over time, the system updates its skills as the project evolves, helping developers write better and more consistent code.
Open → 2610.36188v1

Rivet improves code generation in existing software repositories

Rivet: Utility-Preserving Context Control for Repository-Level Coding Agents

Abstract: Repository-level code generation asks an agent to implement a task requirement inside an existing repository. To produce code that works in the repository, the agent needs repository evidence, such as target locations, helper definitions, caller usages, and configuration constraints, which determine how the code should be located, integrated, and executed. Because the context budget cannot hold the entire repository, existing retrieval, compression, and structure-aware methods select context by task similarity, context length, or structural proximity, but these criteria do not directly measure whether the selected context contains this evidence. As a result, the selected context can appear relevant to the task yet miss the evidence that the implementation depends on. We call this failure mode context utility dilution. In this paper, we propose Rivet, a context control framework that preserves repository evidence for coding agents under a fixed context budget. Rivet, first uses Signal Grounding to select task-related code anchors by fusing lexical, semantic, and structural signals. Utility Linking then expands these anchors through dependency, usage, and configuration relations to collect repository evidence. Finally, Budget Shaping packs the evidence by priority and, when the budget is tight, reduces snippets to signatures or summaries before dropping them. We evaluate Rivet, on two executable repository-level benchmarks, DevEval and RepoExec, with four backbone models, two context budgets (4K and 8K tokens), and eight baselines. Rivet, achieves the highest Pass@1 in all 16 settings and improves over the strongest baseline by 3.73--5.15 percentage points on average. Moreover, GPT-5-mini with Rivet, outperforms GPT-5 without Rivet,on both benchmarks while reducing inference cost by 73.3%--84.5%.

Mon 26 OctSoftware Engineering
The gist
It can be hard for AI coding assistants to add new code to a big software project because they must understand important information like where the code goes and how it connects to other parts. The authors identify that simply picking parts of the project that look relevant can miss this critical information. They created Rivet, a method that carefully picks and summarizes the right clues from the project so the AI can write better code within its limited memory. Tests show Rivet helps AI write working code more often and reduces computing costs.
Open → 2610.34967v1

Large language models explained simply for engineers

Large Language Models: A Plain-Language Overview for Engineers

Abstract: Large language models (LLMs) are neural networks trained on huge text collections to generate text by predicting what comes next. Engineers using ChatGPT and other generative artificial intelligence (AI) tools need a clear picture of what training changes and prompts add. The article starts with the Transformer, which connects words in context, and pre-training, where models learn to predict text. It then explains how fine-tuning adapts models to requests, with reinforcement learning from human feedback using human answer rankings. Next, retrieval-augmented generation (RAG) supplies documents, while chain-of-thought prompting encourages intermediate steps and agents use tools. FlashAttention lowers training and running costs by reducing data movement during attention; quantized low-rank adaptation (QLoRA) saves fine-tuning memory. Readers get a life cycle figure and tables comparing methods and their limits. A short checklist for judging LLM evaluation claims flags test material in training (data contamination) and model-based grading (LLM-as-a-judge). It also covers very long inputs (long context) and hallucination, meaning false or unsupported claims. Check factual accuracy separately from whether an answer sounds helpful.

Sun 25 OctComputation and LanguageArtificial IntelligenceMachine Learning
The gist
Large language models are powerful programs that predict what text comes next by learning from vast amounts of writing. The authors explain key parts of how these models work, like the Transformer architecture and how they're initially trained and later fine-tuned to perform specific tasks. They also describe techniques like using documents to improve answers, encouraging step-by-step reasoning, and using tools to extend capabilities. The paper provides a helpful overview, including how to check if the models’ answers are reliable or just sound convincing.
Open → 2610.34585v1

DoorDash automates removal of obsolete software feature flags

Automating Feature Flag Cleanup at Scale with Multi-Agent LLM Systems: An Industrial Experience at DoorDash

Abstract: Feature flags are runtime configuration mechanisms used for gradual feature rollouts, experimentation, and kill switches. Once a feature is fully rolled out, the flag and its gated code become technical debt and can introduce operational risk through inadvertently enabling dormant code paths. At DoorDash, over 56,000 feature flags span 623 repositories, with 1,659 currently stale as of May 2026, with internal estimates of one to two hours per manual cleanup. Existing rule-based cleanup tools proved insufficient in DoorDash's codebases. Our contribution is an end-to-end automated LLM-based feature flag cleanup system that integrates querying live feature flag rollout data using MCP, Jira ticket processing, and end-to-end pull request creation. Our system successfully produced engineer-approved flag removal PRs for 45 (90%) of the 50 stale flags it was evaluated on, proving the system's operational viability at scale with each cleanup averaging only $4.79 and 13.8 minutes. We evaluated the system's quality and reliability boundary based on success rates, cost per cleanup and time metrics. Furthermore, we discuss our lessons learned when architecting this system, to provide practical guidance for developing similar multi-agent systems.

Thu 22 OctSoftware EngineeringArtificial IntelligenceMultiagent Systems
The gist
Feature flags are like on/off switches for software features, but old ones can clutter code and cause problems. The authors show how DoorDash used advanced language AI programs to quickly and cheaply clean up these old flags across many software projects. The system checks data, reads tickets, and makes code fixes automatically, succeeding in most cases tested. This shows how smart automation can handle complicated coding chores effectively.
Open → 2610.31351v1

Dynamic analysis improves generation of github actions workflows

DynaCI: Generating GitHub Actions Workflows via Dynamic Analysis

Abstract: Continuous Integration (CI) is an essential practice that automatically validates code changes through builds and tests. However, configuring CI workflows remains challenging, as modern software often features complex architectures, numerous dependencies, and support for diverse execution environments. Most CI platforms also require workflows to be written in their particular domain-specific languages. As a result, CI workflows are often error-prone and hard to maintain, especially for those unfamiliar with CI semantics. Prior work on automated CI workflow generation require developers to create specifications that describe their intended workflows using natural language, partial workflows, etc. However, in practice, developers may struggle to articulate requirements and may be unaware of available CI features. To address these limitations, we propose DynaCI, a novel approach that uses runtime information from developers' local environments to generate CI workflows. DynaCI (i) traces the developers' local environment to collect runtime information, (ii) constructs an initial workflow by applying runtime-informed heuristics, and (iii) iteratively refines the workflow via LLM-based Retrieval Augmented Generation (RAG). We evaluate DynaCI on 50 popular Python projects that use GitHub Actions, a popular CI platform, and compare it against the state-of-the-art CI workflow generation approaches: GH-WCOM and LLM4GW. DynaCI outperforms these existing approaches by more than 21.13% in BLEU score and 20.96% in F1 score on average. It also generates workflows without missing functionality in 56% of cases, while requiring fewer manual modifications to repair such exceptions. These results demonstrate that leveraging runtime information substantially improves LLM-based CI workflow generation, and offer a promising direction for broader DevOps configuration automation.

Wed 21 OctSoftware Engineering
The gist
Setting up automated tests and builds for software projects is hard because it involves many complex steps. The authors created a tool called DynaCI that watches what happens when developers run code on their own computers to automatically build these setups. DynaCI then uses a type of AI to improve these setups step-by-step, making them more accurate and easier to maintain. Tests on popular Python projects showed DynaCI makes fewer mistakes and needs less fixing compared to previous tools. This approach could help developers get continuous integration ready without deep knowledge of the underlying systems.
Open → 2610.29890v1

Structured recovery improves fixes in long task stateful language agents

From Trace to Fix: Residual Task State in Long-Horizon Agents

Abstract: Recovering from failures in long-horizon, stateful language agents is challenging because the action introducing a defect, the symptom exposing it, and the persistent state requiring repair frequently diverge. Existing methods largely focus on retrospective failure attribution, leaving sequential diagnosis and state repair poorly integrated under constrained budgets. We introduce TRACE-FIX, a unified recovery framework centered on versioned residual task state. Trace transforms execution logs into a structured dependency-linked ledger, prioritizing unresolved task requirements rather than salient surface errors. Fix formulates recovery as budgeted planning over admissible actions, incorporating outcome forecasting and explicit repair-reserve constraints to prevent premature budget exhaustion. To rigorously decouple localization accuracy from downstream repair capability, we present LongH-Debug, evaluating root-cause attribution, verified-fault recovery, and end-to-end closed-loop execution independently. Our empirical analysis demonstrates that structured residual tracking substantially improves causal localization over raw-trace baselines, while budget-aware sequential planning converts verified fault signals into more reliable task recovery without destructive trial-and-error. Code is available at https://github.com/toffeecat/tracedp.

Mon 19 OctArtificial Intelligence
The gist
Fixing problems in long, complicated tasks that use language-based agents is tricky because the cause of the problem, its symptoms, and the part that needs fixing often don’t match up. The authors developed TRACE-FIX, a system that turns detailed logs into structured records to track what still needs to be done, not just obvious errors. They also created a method to plan repairs carefully within a limited budget to avoid wasting effort. Their tests show this helps find and fix issues more reliably without guessing blindly.
Open → 2610.25776v1

AI agent automates software packaging for reliable builds

Vibenix: Agentic Software Packaging and Maintenance with Nix

Abstract: Implicit dependencies, uncontrolled network access, and other build-environment impurities create intransparent software supply chains. Identifying every dependency affecting a build is a difficult task. Functional package management addresses this through reproducible packaging, ensuring software relies exclusively on declared, hermetically isolated inputs. However, creating build recipes remains complex and time-consuming, posing a significant barrier to adoption for developers and maintainers. We present Vibenix, an open-source AI-powered agent that bridges this gap by automatically generating Nix packaging expressions. Vibenix uses a large language model (LLM) to iteratively refine recipes through a rule-guided build-repair feedback loop. On 466 packaging tasks from our new NixBench dataset, Vibenix produced buildable packages in 95.3% of cases using Claude Haiku 4.5, compared with 5.6% for a state-of-the-art LLM-based single-shot packaging tool. Manual validation of 100 packages revealed that 47% of the buildable packages were initially functionally correct; a post-build refinement process based on the evaluator-optimizer pattern increased functional correctness by 35 percentage points. On our new NixBench-Repair dataset of package maintenance tasks, Vibenix achieved an 87.6% success rate. These results demonstrate how unreliable LLMs can generate rigorously defined, complete, auditable dependency trees for real-world software, reducing maintainer burden while providing a trustworthy foundation for secure software distribution.

Sat 17 OctSoftware Engineering
The gist
Software often depends on many hidden parts which makes building it difficult and unreliable. The authors built Vibenix, an AI helper that writes recipes for a tool called Nix to package software correctly. Vibenix uses an AI model to try building a package, learn from errors, and improve its instructions step by step. It successfully created working packages in over 95% of tests and fixed existing packages with high accuracy, making software building more trustworthy and easier for developers.
Open → 2610.24712v1

Precisecoder improves code debugging by reducing unnecessary edits

PreciseCoder: Faithful and Generalizable Debugging Beyond Unit-Test Rewards

Abstract: Reinforcement learning (RL) from unit tests is the common recipe for post-training code models, but a unit test only asks whether the final program passes. We analyze what this unit-test-only reward teaches a model during debugging and find that, while it raises pass rates, it also teaches the model to edit more and to hallucinate bugs in code that is already correct. We introduce PreciseCoder, a training algorithm that augments the unit-test reward with a signal for semantically verified, localized edits, which correlates with smaller edits and higher pass rates. On 4B, 9B, and 27B models, PreciseCoder reduces the average number of edit lines from 7-15to 3 at a pass rate on par with unit-test-only training. Trained only on single-line bugs, PreciseCoder can generalize to repository-level program repair, where it resolves more issues with smaller patches. Its precision also transfers from weaker to stronger models: given the reasoning trace of a 9B PreciseCoder model, frontier models such as GPT-5.6 Sol and Qwen3.8-Max gain 4.6-12.4 points in edit precision with no additional training. Linguistic analysis of the thinking traces shows that our reward encourages the model to reason about individual lines, name the buggy ones, and form consistent bug hypotheses.

Tue 13 OctSoftware EngineeringArtificial IntelligenceComputation and Language
The gist
Finding bugs in computer programs usually involves testing if the program works after changes, but this can lead models to make many unnecessary edits or wrongly assume bugs exist. The authors present PreciseCoder, a method that encourages smaller, more focused changes that are verified to fix real bugs. It performs well on different model sizes and even helps bigger code models improve without extra training. This approach teaches models to think more carefully about which lines need fixing and why.
Open → 2610.18755v1

EviRepair improves software vulnerability fixes with reliable evidence

EviRepair: Evidence-Driven Agents for Reliable Vulnerability Repair

Abstract: Recent advances in large language models (LLMs) have enabled agent-based approaches for repository-level automated vulnerability repair (AVR), showing promising repair capabilities. However, an important issue has received limited attention: how can we ensure that the evidence generated during the repair process is reliable enough to support accurate vulnerability understanding and patch generation? In this paper, we propose EviRepair, an evidence-driven multi-agent framework for repository-level AVR. EviRepair introduces evidence review into the repair process by enabling agents to collaboratively collect vulnerability evidence, verify evidence consistency, and construct a reviewed vulnerability model for patch synthesis and validation. Extensive experiments on PatchEval and SecBench benchmarks with multiple agent frameworks and LLMs demonstrate that EviRepair consistently outperforms existing agent-based baselines, achieving average improvements of 16.37% and 40.33% in resolved rates on PatchEval and SecBench, respectively. Codes are available at https://github.com/lly-s/EviRepair

Fri 9 OctCryptography and Security
The gist
Fixing security problems in software is tricky because machines need good clues to understand and patch vulnerabilities correctly. The authors created EviRepair, a system where multiple AI agents work together to gather and check information about security flaws before fixing them. By reviewing evidence collaboratively, EviRepair produces more accurate and reliable patches. Testing on standard benchmarks showed it fixes more vulnerabilities than earlier automated methods.
Open → 2610.13491v1

Ai agents modifying code and deployment face integration challenges

Co-Modifying Application Code and Deployment Infrastructure: A Causal Study of AI Agent Pull Requests

Abstract: Current software engineering agents are evaluated predominantly inside isolated sandboxes against synchronous local tests. However, deploying features in collaborative repositories requires modifying continuous integration (CI/CD) pipelines, containers, and provisioning manifests alongside application logic. Because agents lack deployment awareness, attempting cross-domain modifications introduces unverified dependencies into asynchronous integration pipelines. To isolate the causal integration overhead of dual-scope pull requests (PRs)-modifying both application code and deployment infrastructure-we evaluated 26,999 PRs across two large datasets (AIDev v5 and MOSAIC-3M) encompassing six agent architectures and an aligned human baseline. Using a deterministic four-tier file taxonomy and an exact 4D strata-matching framework, we isolated integration metrics under conditional exchangeability while controlling for application volume. We find agents operate primarily as application coders or chore workers, with maintenance tasks capturing 17.70% to 25.41% of agent PRs. Dual-scope modifications represent only 4.48% to 4.55% of agentic PRs, yet trigger bimodal integration outcomes. In the first mode, explicitly reviewed contributions experience protracted delays, adding a median penalty of +13.13 hours and multiplying time-to-merge by 3.24x, driven mechanically by CI/CD and provisioning manifests. In the second mode, maintainer gatekeeping exhibits a structural bypass: even for external contributors lacking repository write access, agent identity reduces the odds of human review by 63.7% compared to human peers. Consequently, 43.1% to 56.2% of dual-scope PRs bypass inspection to merge silently in a median of under 31 minutes. These uninspected modifications compromise build verification, demonstrating the need for deployment-aware benchmarks and strict path-based access controls.

Fri 9 OctSoftware EngineeringArtificial Intelligence
The gist
Software agents that help write code usually work in controlled environments and don’t handle deployment settings well. This study looked at nearly 27,000 code changes to see what happens when agents modify both application code and deployment tools at once. The authors found these combined changes happen rarely but cause two main problems: some changes take much longer to integrate due to extra testing, while others skip important human reviews, potentially letting errors slip in. This shows the need for better tools that understand deployment and stricter control over who can change deployment-related files.
Open → 2610.12849v1

Reusable code components improve building large software repositories

Large-scale Repository Engineering via Agent-Native Reusable Code Primitives

Abstract: Large language models equipped with development environments have moved code generation toward repository-scale construction, yet building complete repositories remains difficult because interacting modules, interfaces, configurations, tests, and dependencies must work together. We introduce Code Primitives, agent-native reusable executable components with interface contracts, dependency closures, validation tests, and provenance. Each primitive uses a resident LLM to assess relevance and adapt its implementation, interfaces, and dependencies to the target repository, and we organize 1,424 validated primitives in CodeFace, a searchable library for repository construction. We introduce LEGO (Large-scale repository Engineering via aGent-native reusable cOde primitives), which activates task-relevant primitives, integrates their adapted implementations with task-specific code while resolving cross-component constraints, and revises the result against executed tests. To measure construction end to end, we build LEGO-REPO, a benchmark of 522 executable reconstruction tasks spanning seven software domains, 22 capability tracks, and five difficulty levels, scored against native test suites between an empty-package floor and original-source ceiling. The strongest of 13 evaluated backbones reaches a delivery score of 0.318 and scores zero on 41.0% of tasks; LEGO improves all 13 by 0.1474 on average and raises GPT-5.6-terra from 0.3180 to 0.5134 (+61.4%). In controlled comparisons, adapted primitives outperform retrieved code supplied as context or vendored unchanged. The effect persists against independent repository agents, across three external benchmarks, and with a disjointly re-mined CodeFace; GPT-OSS-20B for adaptation and diagnosis retains 95.1% of the homogeneous score at 24.0% lower cost.

Tue 6 OctSoftware EngineeringComputation and LanguageComputer Vision and Pattern Recognition
The gist
Building large software projects is hard because many parts need to work well together, like code modules, tests, and dependencies. The paper introduces 'Code Primitives,' smaller reusable code pieces that can automatically adapt to specific projects using a language model. They created LEGO, a system that picks and adapts these code primitives to build a whole repository, checking and fixing errors along the way. Their tests show LEGO improves results over previous methods, making software construction more reliable at scale.
Open → 2610.09079v1

Retrace improves software fix reuse by untangling repair histories

RETRACE: From Entangled Repair Histories to Reusable Experience for CI Repair

Abstract: Large language model (LLM) agents increasingly reuse prior experience, but most approaches assume that problems and solutions are already aligned. Software histories rarely provide this alignment: a pull request (PR) may contain multiple continuous integration (CI) problems, failed attempts, reverted edits, and unrelated changes, obscuring which changes resolve each problem. We present RETRACE, a framework for reconstructing problem-level repair experience from such histories. RETRACE combines an endpoint view that reasons backward from changes retained in the passing revision with a development view that traces repair evolution forward through commit history. CI execution evidence reconciles the two views, and the recovered experience is represented at three abstraction levels, from concrete fixes to transferable repair patterns. For new failures, RETRACE retrieves relevant problem-level experience to guide repair. On CI-REPAIR-BENCH, comprising 565 PR-level repairs from 101 repositories across 12 failure categories, RETRACE improves mini-SWE-agent Pass@1 from 19.6% to 31.9% with MiniMax-M2.5 and from 23.3% to 32.8% with DeepSeek-V4-Flash. On a matched subset, Codex improves from 15.5% to 27.5%. Combining both views consistently outperforms either alone, showing that recovering problem-change alignment enables historical CI repairs to serve as reusable repair experience.

Sat 3 OctSoftware EngineeringArtificial Intelligence
The gist
Fixing software problems can be tricky because developers' past fixes are often mixed up with failed attempts and unrelated changes. The authors present Retrace, a method that separates and organizes these past fixes by understanding which changes really solved each problem. This helps software tools learn from past fixes to better solve new issues. Retrace was tested on many real software fixes and showed it can significantly improve automated repair success.
Open → 2610.04658v1

Improving test coverage reveals hidden flaws in agent verification suites

Discriminating Fixture Coverage in Agent-Infrastructure Verification Suites

Abstract: Invariant suites and runtime monitors increasingly gate agent deployment decisions, and the evidence offered for any particular suite is almost always a single observation: it passes an implementation believed correct and fails one believed broken. We measure what that observation is worth. Applying mutation analysis to an invariant suite for a multi-session agent state-projection layer, we first find that this standard validation certifies a suite in which a first-order mutant removing event-identity deduplication survives every check. We then freeze the repaired twelve-check suite, record its hash, and run it once against ten mutants specified by an adversarial reader who designed none of its fixtures: it kills five. Instrumenting the five survivors against the reference shows they fail in two distinct ways, not one. Three are never activated, because no fixture supplies an input on which the mutated code behaves differently at all. The other two corrupt internal state that no oracle in the suite can observe. The two modes need different repairs, and neither is visible from a pass/fail report. Treating the missing inputs as a coverage question, we enumerate seven discriminating dimensions of the input space, register in advance which are uncovered and which survivors they should explain, and add one fixture per uncovered dimension while reusing the existing oracles verbatim. All five survivors then die, each to the check written for its predicted dimension. We report this as a repair result on the same challenge set rather than a second held-out estimate, and give the artifact, including the frozen hash, the registered predictions, all mutants and the run logs, so the distinction is checkable.

Fri 2 OctSoftware EngineeringArtificial IntelligenceMachine Learning
The gist
Testing whether software works often involves checking if it passes certain tests and fails broken versions. The researchers show that just passing or failing these tests doesn't always catch all the problems. They found that some tests miss important cases where bugs hide because the test inputs don’t cover all possibilities. By adding new, carefully chosen test cases, they made the tests catch all hidden problems. This approach helps make software checking more thorough and reliable.
Open → 2610.02928v1

Terminal agents struggle to detect and fix their own errors reliably

Can Terminal Agents Trust Their Own Verification? Diagnosing and Improving Self-Verification

Abstract: Terminal agents rely on self-verification to assess and correct their solutions as they solve tasks through interaction with command-line environments. Yet how trustworthy such self-verification is remains poorly understood. To investigate this question systematically, we introduce a diagnostic framework that identifies the first complete solution in each trajectory, determines whether it is objectively correct, and uses this ground truth to quantify the agent's subsequent verification and recovery behavior. Applying it to ten terminal agents on TerminalBench2.1, we find that verification is nearly universal after a complete candidate is formed, yet only 61.43\% of incorrect candidates are detected and only 49.36\% of detected errors are successfully repaired. These results show that the main weakness in self-verification lies not in initiating verification, but in detecting and repairing errors. Motivated by these findings, we propose Student-Conditioned Verification Distillation (SCVD), which lets the student first produce a candidate solution and distills a stronger teacher's subsequent verification and recovery from the same interaction context. Across three Qwen3.5 backbones, SCVD improves \textsc{Pass@1} on TerminalBench2.1 by 9.74--16.85 percentage points over the corresponding base models and by 4.49--8.61 points over the standard full-trajectory distillation, while avoiding the pronounced out-of-distribution degradation of full-trajectory distillation on SWE-bench Verified.

Wed 30 SeptComputation and LanguageMachine Learning
The gist
Some AI tools solve problems by trying commands and checking their own work through a computer terminal. The paper shows that these tools often fail to notice when they make mistakes or to fix them successfully. The researchers studied this issue closely using a special test setup and found that while these tools usually try to check their answers, they miss many errors and don't fix half of the errors they find. To improve this, the authors developed a new method that teaches the tools to verify and correct better, leading to more successful problem solving.
Open → 2609.38812v1

Live architecture models track kubernetes deployments for drift detection

Live Architecture Models for Cloud-Native Architecture-as-Code: Early Results from Kubernetes Conformance Checking

Abstract: Infrastructure drift can separate a running Kubernetes system from its documented architectural intent. This paper investigates live architecture models: editable architectural representations connected to selected runtime facts through explicit correspondences and recurring conformance checks. The approach is instantiated through an Archer-specific subset of Kubernetes Deployment Language (KDL) and Archer, a VS Code prototype with synchronized textual and graphical views. Snapshot recovery extracts selected Kubernetes facts into KDL; periodic and on-demand read-only checks report model-cluster inconsistencies without enforcing or repairing deployment state. We assess feasibility on three feature-selected Kubernetes example applications under an author-defined protocol, reporting precision and recall for snapshot recovery and selected inconsistency detection. Recovery scores can be reproduced from saved ground-truth and recovered models; detection uses a documented manual perturbation protocol. The results support feasibility within the evaluated KDL scope. They do not establish comparative superiority, detection of arbitrary production drift, scalability, or developer benefit. Storage coverage and ingress-host representation/default handling remain limitations.

Mon 28 SeptSoftware Engineering
The gist
Kubernetes systems can sometimes run differently than originally planned due to changes or errors, which is called infrastructure drift. The authors explored a way to keep the system’s actual state and its intended design connected using editable models that reflect live Kubernetes data. Their tool periodically checks for inconsistencies without automatically fixing them, helping developers spot where the system drifts from its intended setup. They tested this approach on a few example applications and found it is feasible for certain Kubernetes configurations.
Open → 2609.36148v1

Compiler can access code part but coding agent cannot enforce it

The Compiler May Read It, the Agent May Not: Keeping Part of a Research Code Away from a Coding Agent

Abstract: The compiler must read modules a physics-based solver cannot build without; the coding agent must not read that intellectual property. The harness does not ship that rule. We classified fifteen read routes against a container, permission rules and a sandbox. None of the three can tell which program is reading.

Mon 28 SeptCryptography and SecurityArtificial IntelligenceSoftware Engineering
The gist
Sometimes, parts of computer code need to be hidden from helpers that write or modify code, but still accessible to tools that prepare the program for running. The authors looked at different ways to control who can read these hidden parts and found that common methods like containers, permissions, and sandboxes cannot tell who is reading the code. This means it's hard to protect some intellectual property in code from being seen by automated coding assistants, even though it's needed by compilers.
Open → 2609.35557v1

Planarian improves AI agent state management for better task recovery

Planarian: Managing Agent State with Statepoints

Abstract: LLM agents solve complex tasks by iteratively changing files, invoking local tools, and interacting with remote services, which modifies state across their local environment and remote services. Today, agents and users must manage these changes explicitly, whether reverting exploratory actions or recovering from erroneous ones. Doing so safely requires coordinated actions, yet current agent harnesses lack unified abstractions and mechanisms for managing local and remote state consistently and efficiently. We describe Planarian, an agent runtime with state management that enables agents and users to recover from erroneous actions and explore alternative executions over consistent local and remote environment state. Planarian introduces the abstraction of agent statepoints, which are consistent, restorable point-in-time versions of the environment state. Planarian exposes three state-management primitives to agents and users: (i) snapshot creates a new statepoint spanning local and remote state without requiring external services to support checkpoints: it relies on efficient incremental process and file system snapshotting to capture local sandboxed state, and transparently records compensating actions to undo remote state changes; (ii) rollback restores the environment to a previous statepoint by reverting to a prior local checkpoint and replaying compensating actions for remote state changes; and (iii) fork creates multiple isolated branches from a statepoint, enabling the agent to explore alternatives in parallel. We show that Planarian enables agents to undo mistakes and explore alternatives in parallel, improving task quality by up to 15x, and allows users to recover from erroneous actions with only 3% overhead.

Mon 28 SeptOperating SystemsArtificial IntelligenceCryptography and Security
The gist
LLM agents work by making changes across files, tools, and online services, which can get messy and hard to fix if errors happen. The authors created Planarian, a system that lets these agents save snapshots of their state, undo mistakes, and try out different options easily. Planarian keeps local and remote parts of the agent’s environment in sync and safe, which helps agents and people recover from problems faster. This approach improved task success by up to 15 times while adding little overhead.
Open → 2609.35366v1

Coding agents often duplicate code despite functional success

Do Coding Agents Reuse Existing Code or Reinvent the Wheel?

Abstract: Coding agents are increasingly deployed for iterative development on real repositories, yet existing evaluation barely answers a basic question: \emph{do coding agents reuse existing code or reinvent the wheel?} The question matters: every duplicated implementation is a fix applied twice and agents produce code far faster than humans can audit, so redundancy accumulates unsupervised. Thus, we present \textbf{RepoReuse}, a multi-turn benchmark for auditing code reuse in real repositories, where requirements are revealed turn by turn and the workspace accumulates across turns. It is built by a fully automated pipeline combining AST-based dependency graphs, guided evidence collection, and execution-verified task synthesis, and scales readily to new repositories. Beyond pass rates, we measure the reuse rate together with recall and cross-turn structural redundancy. An audit over 3{,}000 turns shows that agents progressively stop exploring relevant repository code, reuse their own history less even when it is fully in the workspace, and leave duplicated logic in 50.8\% of task chains by turn~5---all while pass rates barely move. Such deficiencies are invisible to pass rates, underscoring the need to evaluate code generation beyond functional correctness.

Mon 28 SeptSoftware EngineeringArtificial IntelligenceComputation and Language
The gist
Coding agents that write software sometimes copy the same code multiple times instead of reusing existing code. The authors built a benchmark called RepoReuse to check if these agents reuse code or reinvent it during multiple steps of software development. They found that agents frequently leave duplicated code even when their code passes all tests, which could cause extra maintenance work. This shows that just checking if code works well is not enough to evaluate coding agents.
Open → 2609.35357v1

Graph guided method improves software environment setup success

Graph-Guided Repository Environment Construction

Abstract: Coding agents now increasingly rely on execution to validate their solutions, making the construction of reliable execution environments a critical enabling capability. However, repository environment construction is challenging because execution requirements are fragmented across repository artifacts and may only become apparent during execution. Existing agent-based approaches address this problem through iterative interaction, but information about the current construction state, including discovered requirements, satisfied and unresolved prerequisites, and their dependencies, can remain distributed across the interaction history. We present Graph2Env, an agent-based approach centered on DepGraph, a typed dependency graph that explicitly represents the environment requirements needed for repository execution, their dependency relations, and their states. Graph2Env uses DepGraph to guide environment construction and continuously refines it with execution feedback, while persisting successful repairs into a replayable construction procedure. The resulting artifacts are then applied in a fresh environment to verify that the constructed environment can be reproduced. We evaluate Graph2Env on a benchmark of 200 Python repositories drawn from RATBench and EnvBench, against a static dependency-inference baseline (pipreqs), three specialized environment-construction systems (Repo2Run, RAT, and SetupX), and two general-purpose coding agents (SWE-agent and Claude Code). Graph2Env achieves an 81.0% Environment Build Success Rate (EBSR) and a 59.3% Environment Setup Success Rate (ESSR), outperforming the strongest baseline by 9.5 and 9.0 percentage points, respectively.

Sun 27 SeptSoftware EngineeringArtificial Intelligence
The gist
Getting software project environments ready to run code properly is hard because requirements are spread across different files and may only show up when testing the code. The authors created Graph2Env, a tool that maps out all the needed parts and their connections in a clear graph to help build these environments step-by-step. By learning from errors during execution and keeping track of what works, Graph2Env builds setups that can be repeated reliably. When tested on many Python projects, it did better than existing methods at preparing working environments.
Open → 2609.33429v1

Sandbox security flaws in ai orchestration expose high risk exploits

Weird Machine Compositors: Exploiting AI Orchestration at the Expression Layer

Abstract: Orchestration platforms secure user-provided expressions through enumerate and block sandboxing: AST rewriting, runtime property blocklists, template sandbox environments. We demonstrate that these sandboxes are weird machines whose instruction set is the underlying language specification, and that the enumerate and block approach is unfixable, following the same trajectory that led to the deprecation of past sandboxing technologies such as Java's SecurityManager and vm2. We validate this claim through three rounds of escalating bypasses against n8n's expression sandbox (three CVEs, two CVSS 9.4, one unauthenticated), and frame these findings within a broader pattern of sandbox failures across the orchestration products category. We identify a trust laundering pattern where orchestration pipelines and applications move attacker controlled input from untrusted to fully credentialed through transformations that strip taint at each level. AI-assisted enumeration accelerates the discovery of these coverage gaps, compressing the timeline between a sandbox's deployment and its compromise. We provide an AST coverage analysis methodology, an accompanying open-source tool, and a defensive playbook that includes policy inversion (allowlist over blocklist) as a structural mitigation.

Sun 27 SeptCryptography and Security
The gist
The paper shows that common security measures used to lock down user expressions in AI orchestration platforms are not effective. The authors demonstrate that these sandboxes behave like 'weird machines' that can be manipulated to bypass restrictions. They found multiple serious vulnerabilities in a popular platform called n8n and explain how attackers can move from untrusted inputs to fully trusted access by tricking these systems. They also show that AI helps attackers find these weaknesses faster. The authors suggest a better defense approach that focuses on allowing only known safe operations instead of blocking certain unsafe ones.
Open → 2609.33413v1

AI code fixes overload review and build systems in big projects

Orchestrating AI-Assisted Code Remediation: Socio-Technical Bottlenecks in a Large Industrial Repository

Abstract: Background: Code degradation in large, long-lived codebases is costly to remediate through manual refactoring and opportunistic clean-ups. LLM-based coding assistants can perform mechanical remediation at scale, but their impact on industrial workflows is underexplored. Objective: We investigate how massive AI-assisted code remediation affects build-on-commit continuous integration (CI), code review, and team coordination in a large industrial repository, and which socio-technical bottlenecks constrain such remediation when source editing becomes cheap through AI assistance. Method: We report on a 15-day exploratory single-case field study in which an experienced developer used a command-line AI coding buddy to remediate widespread issues in a closed-source industrial C++ repository. We triangulate Gerrit metadata with a developer diary and team chat, analyzed through descriptive statistics and qualitative coding. Results: AI-assisted remediation rapidly generated hundreds of commits touching thousands of lines, saturating CI and reviewer attention. Naïve per-file commits overloaded build-on-commit CI; Switching to directory-based batching and capping the number of files per change restored throughput, but still required explicit review solicitation, negotiation of acceptable commit granularity, and iterative follow-up to resolve build and static-analysis failures. Conclusion: When mechanical editing is cheap, CI capacity, review effort, and change orchestration become primary bottlenecks. Sustainable AI-assisted remediation in very large repositories requires deliberate control of commit, review, and CI batch granularity and treating semantic change sets, such as ``fix all instances of warning X'', as first-class units of work that can be sliced differently for developers, reviewers, and CI.

Thu 24 SeptSoftware Engineering
The gist
Fixing lots of code problems by hand is slow and costly. The authors studied how using AI tools to quickly fix code issues in a large industrial project caused new challenges. These fixes created many code changes that overwhelmed automated building and human review systems. The study found that managing how and when code changes are grouped and reviewed is very important when AI makes editing easy. This helps keep the repair process smooth and sustainable in big software projects.
Open → 2609.29172v1

Algebraic architecture theory helps understand software changes reliably

Foundations of Algebraic Architecture Theory: A Rising Sea of Geometry, Transport, Comparison, and Reconstruction

Abstract: AI-generated software changes make it increasingly important to determine what a change preserves, where local consistency fails to extend globally, and which alternatives remain. We develop the foundations of Algebraic Architecture Theory (AAT) from Atoms, typed primitive facts, and Laws, equations that objects must satisfy. A reading specifies what counts as structure and which operations and laws to preserve. The main reconstruction theorem identifies the category of full geometries and all their structure-preserving morphisms with an independently defined category of local models, up to equivalence. Objects are recovered up to isomorphism and morphisms between fixed endpoints uniquely. The theory addresses gluing, diagnosis, transport, classification of changes, and reconstruction. From finite Atom families we construct cores closed under operations and geometries with sites and coefficients. We give conditions under which a Cech obstruction detects the existence of a global state and, through comparison with repair semantics, a global repair. We compare diagnoses and give a finite criterion for uniform invariance given computable finite data. Transport along exact changes has a universal property and commutes with base change on exact pointed pullback squares. Comparisons of routes generated from the same square, finite comparison diagram, and geometry factor into an invertible comparison and an idempotent normalization. We characterize when observations determine comparison preservation and classify compatible lifts. Encodings of lens and protocol semantics preserve and reflect laws and recover semantics-preserving morphisms. Applications classify and count operation-preserving changes and extend morphisms uniquely from finite tables. Corresponding Lean declarations are listed in the appendix.

Wed 23 SeptSoftware EngineeringProgramming Languages
The gist
Software changes made by AI need to be examined carefully to see what parts stay the same and where problems might appear. The authors develop Algebraic Architecture Theory (AAT) to capture the structure and rules that software must follow, helping to reconstruct and compare different versions. Their main result shows how local pieces relate to the whole, making it easier to detect if something breaks globally or can be fixed. This theory also helps classify changes, diagnose issues, and transport information accurately between versions.
Open → 2609.27638v1

Benchmark evaluates AI agents configuring software deployment environments

FDE-Bench: Evaluating LLM Agents for Deployment Environment Configuration

Abstract: Deployment requires an agent to turn application code into a running system whose services connect, become ready, and remain observable. FDE-Bench evaluates this capability with 136 deployment-configuration tasks spanning Docker images, multi-service Compose stacks, and Kubernetes, in greenfield and diagnose-and-repair modes. Agents submit declarative artifacts that are collected, rebuilt, and redeployed in a pristine environment. Four gated binary check layers measure build, readiness, behavior, and conformance to the deployment specification, using programmatic checks without an LLM judge. A four-arm release gate requires a resolving reference solution and rejects tasks solved by do-nothing, specification-transcription, or generic-stub submissions. The released check annotations expose the link between 2,145 checks and their specifications, including seven documented gaps. Three additional adversarial strategies test shortcuts in the grading signals; none resolves any of the 135 tasks they cover, while a vacuous health probe passes readiness and exposes the need for downstream checks. On the 136-task evaluation grid, seven language models from four providers use the same four-tool scaffold and resolve 52.9-75.0 percent of tasks. The three zero-intelligence floors resolve none and reach a mean Deployment Score of at most 0.44. Readiness is the largest failure stage, accounting for 110 of 313 unresolved episodes. Mean resolution rate is 30.7 percentage points higher on the repair task group than on the disjoint greenfield group, with a positive gap for every model; ten tasks resist all seven. In a 25-task case study, one practicing engineer directing Claude-Sonnet-5 resolves 92 percent against 72 percent for the autonomous baseline. FDE-Bench links deployment success and failure to artifacts that can be inspected and replayed.

Wed 23 SeptSoftware EngineeringArtificial Intelligence
The gist
Setting up software so it runs properly and stays working can be tricky. The authors created a test suite called FDE-Bench that checks how well AI systems can turn code into working software setups using technologies like Docker and Kubernetes. Their tests include building, starting, and checking if the software works as expected, without using AI to judge the results. They found AI can solve many but not all deployment tasks, especially struggling with readiness checks. The benchmark also helps engineers understand and improve deployment processes by linking successes and failures to specific configuration artifacts.
Open → 2609.27571v1

Kubernetes security misconfigurations identified and fixed with AI models

Kubernetes Misconfigurations in the Wild: Taxonomy, Evolution, and Automated Repair with Large Language Models

Abstract: Kubernetes is widely used to orchestrate cloud-native applications, yet its declarative configuration model often introduces security misconfigurations that threaten system reliability. Despite available detection tools, misconfiguration patterns and scalable remediation remain insufficiently understood. This paper presents an empirical study of Kubernetes security misconfigurations based on 2,662 developer-reported Stack Overflow issues. We derive a taxonomy of recurring security weaknesses across configuration objects and categories. We analyze severity variations and investigate how misconfigurations evolve between incubator and stable project stages. Findings show that while some operational issues decrease as projects mature, critical security misconfigurations often persist or reappear. We then evaluate Large Language Models (LLMs) for automated remediation under progressively enriched contextual conditions. Contextual grounding improves correction accuracy, with the best standalone model achieving 89.06%. To enhance structural correctness and schema compliance, we introduce Kubecurity, a schema-guided validation framework based on official Kubernetes specifications. Combining contextual LLM reasoning with deterministic schema enforcement achieves 98.50% correction accuracy while substantially reducing newly introduced misconfigurations. This work advances the understanding of Kubernetes security misconfigurations and demonstrates a hybrid approach to more reliable automated remediation.

Tue 22 SeptSoftware Engineering
The gist
Kubernetes is software that helps run applications in the cloud, but setting it up can cause security mistakes that make systems unsafe. The authors studied many real problems reported by developers online to find common security errors and see how they change over time. They tested large language models (AI that understands and writes code) to automatically fix these mistakes and created a tool to check fixes against official rules. Together, the AI and the tool corrected nearly all errors while avoiding new ones, helping make Kubernetes setups safer.
Open → 2609.27030v1

Category-aware training improves software engineering AI performance

One to More, More to One: Category-Aware Iterative Expert Training for Software Engineering Agents

Abstract: Repository-level software engineering (SWE) comprises heterogeneous task categories, whose progress under pooled agentic reinforcement learning can be uneven: gains in some categories coincide with regressions in others, while aggregate resolution obscures these changes. Motivated by this category see-saw, we develop a category-aware expert-training and policy-integration framework. Executable task construction and SWE Labeler, an evidence-grounded multi-axis labeling system, organize the training pools. Initial category-specific RL improves average training success while leaving uneven instance-level progress, motivating explicit consolidation of successful behavior and policy-adaptive task selection. Same-origin category experts alternate long-horizon Agentic-miniRL with Refresh-Repair-Expand (RRE): the updated policy refreshes instance mastery, reuses its own verified successful trajectories for Repair SFT, and reselects tasks for further RL. Label-routed multi-teacher on-policy distillation (MOPD) consolidates the experts into one deployable student, with ReLU-gated reward extrapolation keeping only each teacher's improving direction over the reference. Expert training and policy integration require no external model to provide solution trajectories or action targets. We evaluate Pooled RL and Balanced RL, expert development, and single-model integration through aggregate and per-category resolution, the minimum category lift over each joint-RL baseline, and expert-gain recovery. The final MOPD policy achieves mean resolution of 58.04% on Pro-618 and 59.00% on SWE-bench Multilingual, improving over the base model by 5.39 and 2.78 percentage points, respectively.

Sun 20 SeptSoftware EngineeringComputation and LanguageMachine Learning
The gist
Software engineering tasks can be very different from each other, and when AI agents are trained on many tasks at once, some tasks improve while others get worse. The authors created a way to train AI experts for specific categories of software tasks, then combine their knowledge into one smarter AI. This method helps balance improvements across all software tasks and avoids losing progress on any one type. Their experiments show that this approach makes the AI better at solving diverse programming problems than previous methods.
Open → 2609.23377v1

LLM program repair results need clear experiment details

Experimental Settings in LLM-Based Program Repair: A Study of Inputs, Tool Access, Feedback, and Validation

Abstract: Evaluations of automated program repair (APR) systems commonly report the benchmark, the number of repaired defects, and the tests used for final patch validation, but these items no longer fully specify the repair task presented to a system. Recent LLM-based systems differ in the information supplied before repair, the repository and testing operations permitted during repair, and the feedback returned after unsuccessful attempts, allowing the same benchmark to instantiate substantially different repair tasks ranging from localized patch generation to repository-level diagnosis and iterative repair. We present a framework for explicitly specifying the experimental settings associated with reported APR results. We analyze reported experimental settings from systems evaluated on Defects4J and SWE-bench and characterize each result by its task unit, fault-localization assumptions, initial input, tool access, repair-time feedback, final validation, and resource budget. Our analysis shows that benchmark identity alone is insufficient to reconstruct the evaluated task or determine the appropriate scope of comparison across reported repair rates. We therefore introduce a machine-readable schema for specifying each experimental setting to improve reproducibility and make the scope of cross-system comparisons explicit.

Wed 16 SeptSoftware Engineering
The gist
Fixing computer programs automatically with AI models can be done in many ways, but how the repair task is set up affects the results a lot. The authors looked closely at how different studies provide input data, allow tool use, show feedback, and check fixes, finding these factors are often unclear. They created a system that clearly describes these settings so others can understand and compare results better. This helps make research on AI program fixing more repeatable and fair.
Open → 2609.17993v1

Software development adapts itself as projects and conditions change

Rethinking Software Development as a Self-Adaptive Socio-Technical System

Abstract: Agentic AI is expanding across various software engineering activities, yet human-agent organization is typically treated as fixed, which is problematic because development configurations may become suboptimal as development context evolves. We propose viewing software development as a self-adaptive socio-technical system in which both the software project and the development configuration (participants, responsibilities, authority, information, and verification) adapt as goals, evidence, uncertainty, and risk evolve. This yields two coupled forms of adaptation: evolving the software and reconfiguring how subsequent engineering is performed. We illustrate the idea with a proof of concept, outlining challenges and future plans.

Fri 11 SeptSoftware Engineering
The gist
Software development teams usually follow a set plan for who does what, but this plan can stop working well when things change. The authors suggest thinking about software development as a living system that changes both the software and how the team works as new goals, information, and risks appear. This means the people and their roles in the development can adjust alongside the software itself. They provide an initial example of how this idea could work and discuss challenges ahead.
Open → 2609.13461v1

Reinforcement learning method improves code generation tasks at test time

Entropy-Regularized Rank-Masked Policy Optimization for Test-Time Reinforcement Learning in Code Generation

Abstract: Existing methods for test-time reinforcement learning (TTRL) derive rewards from answer-level self-voting on unlabeled test-time tasks with canonical answers, but this breaks down for code generation because programs cannot be compared by surface form and therefore do not directly provide a usable training signal. To make TTRL applicable to code generation, we propose probe-driven TTRL, which constructs output-free probe inputs from the problem statement, executes candidate programs on these probes, and defines a Probe Consensus Reward (PCR) from the resulting behavioral agreement. PCR provides a behavioral training signal for open-vocabulary programs, but it is not a fully reliable verifier and remains susceptible to reward hacking through spurious consensus. We therefore introduce Entropy-Regularized Rank-Masked Policy Optimization (ERPO), which converts low PCR into conservative negative updates through rank masking and controls policy drift with an entropy ceiling. On coding benchmarks, ERPO substantially improves pass@1 and pass@k in both in-domain adaptation and zero-shot transfer.

Tue 8 SeptMachine LearningComputation and Language
The gist
Generating computer programs is hard because small changes can make big differences, so usual techniques to check answers don't work well. The authors created a way to test programs by running them on special test inputs made from the problem description and seeing if they behave similarly. They use this behavior as a kind of reward to teach the system better. To avoid tricks and incorrect learning, they use a method that carefully adjusts training updates and keeps the program's guessing patterns diverse. Their approach improves the quality of generated code in several tests.
Open → 2609.09135v1