Papers for

software reliability engineers

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

DataTrace benchmarks evidence-based root cause diagnosis in databases

DataTrace: Benchmarking Data Agents for Evidence-Grounded Root-Cause Localization

Abstract: Database agents diagnose pipeline failures, but a correct root cause is trustworthy only when it is supported by evidence the agent inspected. Existing benchmarks usually grade the final query or answer but fail to distinguish an evidence-based diagnosis from an unsupported correct answer. We introduce DataTrace, a benchmark for evidence-grounded root-cause localization over typed graphs of database artifacts. DataTrace uses controlled fault injection to derive root causes and evidence paths within 83 runnable environments without human annotations, and scores root-cause correctness and coverage of the supporting path separately. We also introduce CoSE, a generator-solver protocol in which an external verifier admits reproducible faults and checks both the submitted answer and the inspected evidence. Our evaluation shows that seven proprietary models achieve average root-cause accuracy 48%, with a mean per-model conditional unsupported rate of 18%. Fine-grained analysis shows that this gap varies sharply with fault mechanism, dependency distance, and evidence modality. After in-loop training on a matched held-out batch, the share of grounded correct diagnoses rises from 83.7% to 94.6%, while root-cause accuracy rises from 29.2% to 36.7%. These results show that while final-answer accuracy can hide unsupported diagnoses, evidence coverage provides an actionable signal to improve database agents. Code and data are available at https://hkustdial.github.io/datatrace/.

Fri 13 NovDatabases
The gist
When databases break, it's important not only to find the problem but to show the exact evidence behind the diagnosis. The authors introduce DataTrace, a tool that creates test scenarios with known failures and checks if database agents can find and support the real cause. They show current models often give right answers without proof. DataTrace reveals where these models fall short and helps improve them by focusing on evidence, making database troubleshooting more reliable.
Open → 2611.18926v1

Managing language model errors improves autonomous system decisions

Maximizing LLM Utility While Managing LLM Fallibility

Abstract: Large language models (LLMs) can supply autonomous systems with plans, predictions, and judgments, but these contributions are sometimes wrong. This review asks how a system can maximize an LLM's utility while managing its fallibility. Our answer is to adapt the system's reliance on the LLM as evidence about the LLM's performance accumulates. Systems built from imperfect components, such as noisy sensors and faulty software, have long worked in this way, and the classical methods they use can be applied to LLM-based systems as well. A system can change its reliance on an LLM in several ways, depending on the LLM's role and on the other components in the system. Each change needs an estimate of the LLM's reliability, and the LLM's role and the kind of contribution it makes determine what to estimate. Classical methods from decision theory, estimation, and online learning then turn these estimates into reliance decisions. These decisions can reduce how often LLM errors affect the system. However, some errors matter even when they are rare. An independently verified mechanism can prevent such errors from being executed. Without one, adapting reliance using reliability estimates can still keep their risk within a bound. Studies on adapting reliance on LLMs are emerging but limited, and the reviewed comparisons suggest that adapting reliance can improve system performance.

Thu 12 NovRobotics
The gist
Large language models (LLMs) can help computers make plans and predictions, but sometimes they make mistakes. The authors explain how systems can watch how well these models are doing and change how much they trust their advice over time. By doing this, computers can use the models’ help while reducing the chances that their errors cause problems. The authors also discuss ways to stop the worst mistakes from causing damage, either by independently checking outputs or by keeping risks within limits.
Open → 2611.17020v1

Unsupervised anomaly detection methods vary widely by log data and preprocessing

Fast Unsupervised Anomaly Detection Methods on Software Logs - An Empirical Study

Abstract: Anomaly detection on software logs ensures system reliability through failure identification, root cause analysis and fault localization. While supervised models show excellent performance, in a practical setting there rarely are labeled training data available. This calls for the use of unsupervised models. We study effects of data preprocessing, anomaly detection method, and the dataset. For preprocessing, we examine three options: The use of anomaly-free training data, using randomized as opposed to chronologically selected training data, and the use of a custom loader to extract log message from the log line. Regarding anomaly detection we use four models (KMeans, IsolationForest, RarityModel, Out-Of-Vocabulary Detector) and three representations (Words, trigrams, and parsed events). We evaluate these approaches on seven datasets: BGL, HDFS, Hadoop, Spirit, Thunderbird, Liberty, and LO2Token. We run each combination of the selected factors and measure performance with AUCROC and F1-score metrics. Then we utilize a generalized linear model to establish relationships between the factors. Furthermore, we address execution times. Our evidence reveals large variability across all datasets, preventing us from making recommendations that apply universally. For instance, log parsing enhances performance on some datasets but reduces it on others. The evidence suggests that using anomaly-free data for training unsupervised models can improve anomaly detection performance, though this requires extra effort to identify such data.Previous work has argued that shuffling data leads to unrealistic model performance (data leakage). Yet, our results show this does not hold for the unsupervised models we tested, while it remains true for supervised models in the same setup. For fast execution of the pipeline, use word representation and avoid IsolationForest model.

Thu 5 NovSoftware Engineering
The gist
Detecting unusual problems in software logs helps keep systems running smoothly. The authors studied different ways to detect these problems without needing labeled examples. They found that performance varies a lot depending on the dataset, how the logs are prepared, and which detection method is used. For some datasets, parsing logs into events helps, while for others it does not. Using clean, anomaly-free data for training can improve detection but requires extra effort.
Open → 2611.07760v1

Simple large language models diagnose microservice failures better than complex agents

Free to Look, Free to Reason: Simple LLM Harnesses Outdiagnose Hand-Crafted Root Cause Analysis Agents

Abstract: Root cause analysis (RCA) of microservice failures has attracted many LLM agents whose investigation is hand-crafted, through standard operating procedures, fixed stages and role agents. Each is evaluated on its own data against few baselines, so it is unknown whether this design beats letting a capable model investigate on its own. We compare three simple harnesses that prescribe none of the model's calls (one call over an evidence package, a SQL agent and the Codex coding agent) with five published LLM RCA agents and with statistical and causal methods on RCAtlas, 90 validated cases from eight public datasets, across six models and three reasoning efforts. The simple harnesses are the most accurate (PR@1 65-77% on three mid-tier models), ahead of the published agents (16-53%) and the non-LLM methods (6-39%), on every model on which both families were run. The published agents fix from a tenth to all of their model calls in the program, and a case study shows how these prescriptions skip the decisive evidence. A stronger model mainly improves reliability, a multi-turn harness mainly improves aptitude, and more reasoning effort changes accuracy by a few points at most while multiplying the time by up to seven. Among runs more accurate than the best non-LLM method, every Pareto-optimal run in cost or time uses a simple harness.

Mon 26 OctArtificial IntelligenceDistributed, Parallel, and Cluster ComputingSoftware Engineering
The gist
Figuring out what caused a software failure in complex systems called microservices is hard, so some computer scientists build step-by-step programs to guide AI models in finding the cause. This paper shows that just letting simpler AI models freely investigate the evidence outperforms these complicated, hand-designed processes. The authors tested their approach on many real-world cases and found it was more accurate and often faster. They also discovered that improving the AI model itself helps reliability, while letting it do multiple reasoning steps improves its skill level.
Open → 2610.35767v1

Scale does not improve fault tolerance in tool-calling LLM agents

Tool-Calling LLM Agents Under Realistic MCP Faults: Scale Does Not Purchase Fault Tolerance

Abstract: LLM agents increasingly rely on external tools exposed via the Model Context Protocol (MCP), yet current benchmarks assume tools respond correctly and on time, leaving agent resilience to realistic tool failures untested. We introduce MCP-Injector, a drop-in, protocol-aware fault-injection middleware that interposes on the MCP tool-call path with controlled persistence (permanent, transient, intermittent) and seedable schedules. Its fault model is calibrated to the observed incident distribution (failure mode x persistence) of the public MCP ecosystem, mined from 57,240 issue and pull-request threads across 1,774 open-source MCP repositories. In a 1,350-cell experiment crossing two backbone sizes (20B, 120B), three agent stacks (custom multi-round plan-and-execute, OpenAI Agents SDK, LangChain), and three fault injection conditions, we find that faults drawn from this permanent-heavy stress profile cause severe quality degradation (task-completion score falls by 60%, from 8.08 to 3.23 out of 10), and that scale does not purchase fault tolerance for the two gpt-oss backbones we test: the 120B retains no larger share of its fault-free score under faults (resilience-ratio model, $p = 0.96$), and its completion rate under faults is, if anything, lower (91.1% vs. 95.6%; difference 4.4 points, 95% CI $[-0.2, 9.3]$). Agent stacks differ in how well they absorb faults, which suggests that framework-level error handling matters more than model scale. These results demonstrate that capability benchmarks alone can mask operational brittleness, motivating protocol-aware fault injection as a complementary evaluation dimension for tool-calling agents.

Fri 23 OctSoftware EngineeringArtificial Intelligence
The gist
Large language model (LLM) agents use external tools to complete tasks, but usually these tools are assumed to work perfectly. The authors studied what happens when these tools fail in realistic ways, like responding late or incorrectly. They built MCP-Injector to simulate such faults based on real issues from many open-source projects. Their experiments show that bigger model size does not make agents more resilient to these tool failures, but the software frameworks handling the tools do matter. This means testing LLM agents only on ideal tool behavior can hide serious weaknesses.
Open → 2610.32555v1

Agent design adapts safely amid changing conditions with lower costs

Conservative Meta-Agent Search: Architecture Adaptation under Non-Stationarity

Abstract: Automated agentic-system design is typically evaluated on static task suites with immediate scalar rewards. Deployed systems face persistent shift, delayed dependent feedback, correlated component failures, and costly architecture changes. We formalize meta-agent architecture search as sequential incumbent replacement with switching costs and hard risk constraints and introduce Conservative Meta-Agent Search (CMAS). CMAS retrieves archived architectures and proposes typed graph edits, then scores each against the incumbent by paired shadow replay under common random numbers, with interventional component credit. Promotion requires an empirically calibrated, variance-penalized simultaneous score above a pre-specified margin, and is followed by a simulated canary stage and pre-specified rollback. Since backtests cannot certify meta-level skill, we introduce MetaInvest-Bench, a regime-switching simulator whose specified worker competence makes a surrogate-optimal architecture computable, with leakage-controlled replay, adversarial governance tracks, and five negative controls. Financial streams serve as a risk-sensitive testbed for meta-level generalization; we claim no alpha. Across 96 streams, CMAS cuts normalized mean surrogate dynamic-oracle regret from 0.116 to 0.104 and raises held-out utility from 0.44 to 0.48 over the strongest adaptive baseline; when extra workers add no value, it contracts to near-single-agent teams. Its deployment inference cost is ~6% lower, though its search-inclusive token cost is the highest of the methods compared. The observed per-update false-promotion rate is 3.4% against a nominal 5% target, and 4.8% in a sealed rerun. The calibration is empirical: it concerns an update-level replay statistic on an author-designed benchmark, and guarantees neither full-utility improvement, nor absence of simulated canary-stage harm, nor sequence-level error control.

Thu 22 OctArtificial IntelligenceMachine LearningMultiagent Systems
The gist
Designing automated agents that work well over time is hard when task conditions keep changing. The authors offer a method called Conservative Meta-Agent Search (CMAS) that carefully tests new agent designs against current ones before making changes. They use a special simulator to measure performance and control risks in these updates. Tests show CMAS leads to better agent performance on financial data streams while reducing deployment costs and keeping error rates low.
Open → 2610.31436v1

Post incident software changes take longer and focus on server setup

Characterizing Post-Incident Software Evolution

Abstract: BACKGROUND: After a software incident, organizations attempt to improve the reliability of their software systems by conducting post-incident analysis and identifying actions to prevent or mitigate similar future incidents. Despite the importance of this activity, there is a limited understanding of the nature and characteristics of the software evolution triggered by the analysis process. AIMS: This study seeks to bridge this gap by providing the first empirical characterization of post-incident software evolution. We aim to establish an understanding of how these changes manifest, specifically focusing on how their characteristics (such as effort, location, and timing) differ from standard maintenance and evolutionary changes. METHOD: We conducted a mixed-methods analysis of a 10-year longitudinal dataset from Wikimedia, examining 1,302 tasks and 2,203 code reviews. We categorized these changes across five dimensions: target, effort, abandonment rate, prioritization, and time-to-resolve. RESULTS: Our results show that the post-incident evolution is distinct from other change types. Post-incident changes are more time-consuming, with a median completion time over twice that of other changes and are disproportionately concentrated in a server configuration repository. CONCLUSION: By identifying a few of the characteristics of post-incident evolution, this work provides a foundation for future research into reliability engineering and learning from failure. We highlight further avenues for exploration necessary to fully understand post-incident evolution. Lastly, our work enables researchers to move beyond high-level analysis and begin developing specialized methods and tooling tailored to the nature of post-incident evolution.

Mon 19 OctSoftware Engineering
The gist
When something goes wrong in software, fixing it often involves a special kind of change called a post-incident update. The authors studied Wikimedia’s software over 10 years and found these fixes take more than twice as long as usual changes. They also tend to happen mainly in the parts of the system that manage server configuration. This new understanding helps software teams learn better ways to improve software reliability after problems.
Open → 2610.27380v1

Skill and policy changes cause self-learning agents to fail over time

Evolving Together, Failing Apart: The Skill-Policy Co-evolution Trap in Self-Improving Agents

Abstract: Skill-policy co-evolution, in which an agent's policy model and its skill bank improve each other during training, is a promising paradigm for self-improving agents. Current evaluations score the co-evolved policy on the final bank it trained with (the delivered bank). After deployment, however, skills may be rewritten, added or removed more frequently than the policy model is retrained, leaving the policy to operate on a drifted skill bank that neither the training objective nor existing evaluation protocols address. In this paper, we first establish, through theoretical and empirical analysis, that the fragility hidden by this blind spot is not incidental. The acceptance rule these methods share (cooperative acceptance) ties a policy's competence to the bank it co-evolved with, leaving it fragile once skills drift, regardless of the implementation; empirically, sensitivity to skill drift grows as co-evolution training proceeds. We therefore introduce the CO-evolution Drift Audit (CODA), a deployment-oriented evaluation protocol that simulates skill drift and, alongside success rate under worst-case drift, introduces two new metrics: a consistency rate under meaning-preserving edits and a resistance rate to corrupted skills. Across four co-evolution methods and five benchmarks (six settings), worst-case success is 5-13 points below the peak measured for each delivered pair, erasing 52-93% of the delivered bank's gain over the no-skill baseline. CODA thus reframes the target of co-evolution, from peak performance on the co-evolved bank to the ability to harness skills however they evolve, and provides a shared standard against which future methods can be audited.

Wed 14 OctArtificial IntelligenceComputation and Language
The gist
When AI agents learn by improving their skills and decision rules together, they can perform well at first. But if their skills change after training without updating their decision rules, their performance often drops a lot. The authors found that this problem is built into how these agents work, and it gets worse the longer they train together. They created a new way to test agents by simulating how skill changes affect them, aiming to better measure real-world performance after deployment.
Open → 2610.20845v1

Improving safety guard outputs to better match application parsing needs

REA: From Correct LLM Judgments to Safe Decision Delivery

Abstract: Generative safety guards must classify requests correctly and return outputs that applications can parse. Reasoning guards can improve classification accuracy while producing outputs that applications cannot parse. When applications cannot extract a correct prediction, fallback handling can allow harmful requests or block safe ones. We introduce Resolve--Express--Act (REA), a framework that traces judgments through parsing and fallback to application actions. Experiments on classifiers and safety guards show that correct predictions can appear in incorrectly formatted outputs. Parsing the safety verdict preserves the reasoning guard's accuracy advantage, while stricter explanation requirements reject additional correct predictions. Accepting formatting variations recovers most rejected correct predictions. Held-out tests confirm that output requirements and generation limits affect whether correct predictions can be parsed. In our fail-open replay, matching the parser to the output and fixing formatting reduced the fraction of harmful requests allowed despite correct guard judgments from 95.5% to 0%. A fixed-output comparison shows that model-based extraction can preserve guard verdicts, but is not automatically reliable; deterministic normalization already repairs the observed native-format failures. Fallback replay shows that the same correct prediction can lead to different safety actions depending on the parser and fallback policy, including allowing requests the guard correctly identifies as harmful.

Mon 12 OctArtificial Intelligence
The gist
Safety systems that judge whether requests are harmful sometimes give answers in formats that other programs can’t understand. The authors introduce a way to trace how these judgments are processed, showing that sometimes correct safety warnings get lost because of formatting. By fixing how outputs are formatted or parsed, the researchers greatly reduced cases where harmful requests were mistakenly allowed. This helps make AI safety systems more reliable when used in real applications.
Open → 2610.17473v1

Runtime auditor improves failure prediction in multi-agent systems

Dream2Discern: Learning from Hard Latent Variants for Runtime Failure Prediction in Multi-Agent Systems

Abstract: Reliable deployment of LLM-based multi-agent systems calls for runtime auditors that predict terminal outcomes from observed execution prefixes. These auditors typically learn from a limited corpus of trajectories with verified outcomes. Expanding this corpus requires costly execution and verification, yet broader failure coverage alone provides no explicit criterion for identifying prefixes that the current auditor finds difficult to classify. To this end, we propose Dream2Discern, a runtime auditor trained through implicit augmentation guided by its own prediction loss. During Dream, we select perturbations of the auditor's representation operators to increase prediction loss on recorded prefixes, yielding hard latent variants. During Discern, the auditor learns to predict execution outcomes from these variants using the original labels. The variants are regenerated as the auditor learns, turning a limited corpus into a renewable source of hard examples and requiring no additional trajectory collection or outcome verification. Extensive evaluations on AFTraj-2K and StepShield show that Dream2Discern outperforms competitive failure auditors and achieves over 90% trajectory-level weighted accuracy on both benchmarks while balancing prediction accuracy and timeliness. Project page: https://zbox1005.github.io/dream2discern-project/

Mon 12 OctMachine LearningArtificial IntelligenceComputation and Language
The gist
Predicting when complex groups of AI agents will fail during their tasks is important but difficult because verifying failures is costly and limited by available data. The authors propose a new training method called Dream2Discern, where the system generates challenging new training examples from existing data by tweaking its internal reasoning steps to increase prediction errors. This creates a renewable set of hard cases that help the system learn better failure prediction without collecting more data. Tests show this method improves accuracy and speed in predicting failures across benchmark datasets.
Open → 2610.16071v1

LLM agents improve resilience in high performance computing applications

LLM Agents as Resilience Engineers for Scientific Applications

Abstract: Efficient checkpoint/restart support is essential for resilient HPC scientific applications, but implementing it requires substantial expertise: developers must identify recoverable state, choose globally consistent checkpoint points, and preserve application invariants during restart. We study whether frontier LLM coding agents can automate this process. We build a benchmark suite of 16 MPI applications spanning diverse domains, code sizes, and critical-state structures, and evaluate them with a no-human-in-the-loop generate--validate--revise pipeline for checkpoint/restart synthesis. Across the benchmark, the pipeline produces 41 working resilient implementations. Our results show that agent-driven resilience engineering is practical when critical state is visible or accessible through coherent abstractions: successful runs finish in under one hour on average, consume about 15M tokens, and produce implementations with negligible failure-free overhead and recovery efficiency comparable to human-written code. However, modularized and fragmented state remains a major limitation, with some failed attempts consuming over 100M tokens and 300 minutes without producing a working implementation.

Thu 8 OctDistributed, Parallel, and Cluster Computing
The gist
High-performance computing (HPC) applications need a way to save their work regularly so they can recover quickly from failures. This saving and recovery process is hard to implement because programmers must carefully identify and manage the important parts of the program's state. The authors tested whether advanced language models (LLMs) could automate this task. They found that LLMs can successfully generate checkpoint and restart code in many cases within about an hour, especially when the program’s important data is in clear and accessible structures. However, they struggle when the important data is scattered or complex.
Open → 2610.11260v1

Hidden signals reveal ai agent deception before it happens

Before Agent Tells The Lie: Has Deception Already Been Represented?

Abstract: Large language model (LLM)-based agents can exhibit deceptive behavior during task execution, including hiding failures, fabricating results, or falsely signaling task completion. Existing monitoring approaches mainly detect deception after it appears in observable actions or outputs. In this paper, we investigate whether deceptive behavior can be predicted from an agent's internal representations before it becomes externally visible. We frame deception monitoring as a trajectory-level representation analysis problem and align agent trajectories around key decision points. Using hidden states extracted before these points, we show that future honest and deceptive outcomes can be reliably distinguished, with predictive signals remaining detectable several model calls before the final decision. We further characterize the temporal evolution of these signals: deception-related representations are weak early in execution but become increasingly identifiable as trajectories progress, while transferable structure can emerge before the strongest decision-adjacent signals appear. Finally, we intervene on the identified honest-deceptive representation directions during inference and find that activation steering reduces downstream deceptive behavior, suggesting that these representations influence agent decisions. Our findings indicate that agent deception is an evolving internal process that can be detected and potentially mitigated before it is expressed externally.

Mon 5 OctComputation and Language
The gist
Sometimes AI systems called large language model agents can lie or hide mistakes when doing tasks. This paper shows that the AI’s own internal thought process contains clues about whether it will be honest or deceptive before the lies actually appear in its actions. The researchers found that by looking at the AI’s hidden reasoning moments before decisions, they can predict deception early and even steer the system to be more honest. This means AI lying may start inside the model before anyone sees it, and catching it sooner is possible.
Open → 2610.06576v1

Gpu architecture causes different software fault results

BitIR: Cross-Architecture Fault Injection for Resilience Analysis of Heterogeneous GPU Applications

Abstract: Modern GPU-based HPC systems rely on heterogeneous vendor stacks, yet resilience studies are largely limited to single architectures, leaving it unclear how faults behave across different GPU backends. We present \emph{BitIR}, a cross-architecture fault injection framework that injects deterministic single-bit faults at the LLVM IR level, ensuring semantically equivalent perturbations prior to backend lowering and enabling direct cross-vendor comparison across NVIDIA, Intel, and AMD GPUs. We evaluate BitIR on three production supercomputers -- Polaris (ALCF, NVIDIA A100), Aurora (ALCF, Intel GPU Max 1550), and Frontier (OLCF, AMD Instinct MI250X) -- representing the full spectrum of current leadership-class GPU architectures. Using representative heterogeneous benchmarks, we conduct large-scale injection campaigns across all three systems and classify outcomes into masked results, silent data corruptions (SDCs), and failures. Our results show that identical faults produce markedly different behaviors across vendors: Intel most often masks faults but exhibits more hangs when faults escape masking, NVIDIA exposes more detectable hard failures, and AMD alternates between SDC-dominant and failure-dominant behavior depending on the benchmark and fault site. These findings demonstrate that resilience is not backend-invariant, underscoring the need for backend-aware and workload-aware fault mitigation.

Fri 2 OctDistributed, Parallel, and Cluster Computing
The gist
Computer systems that use GPUs from different makers may behave differently when tiny errors occur during calculations. The authors created BitIR, a tool that introduces the exact same small error into programs before they get customized for different GPUs. They tested this on three top supercomputers with GPUs from NVIDIA, Intel, and AMD, finding that errors caused very different outcomes on each one. This means fixing errors and making software reliable needs to consider the specific GPU being used.
Open → 2610.04037v1

Compound AI systems improve reliability with failure patterns and fixes

Compound AI System Reliability: A Failure Taxonomy and Resilience Pattern Catalog from 150 Production Incidents

Abstract: Deploying compound AI systems reliably and safely requires understanding failure modes that emerge at component boundaries, not within individual models. Cascading errors propagate across component boundaries, silent quality degradation evades standard monitoring, and coordination failures yield incorrect collective behavior from individually correct parts. We analyze 150 production incident reports from open-source compound AI projects and anonymized enterprise deployments to construct a taxonomy of 23 failure modes organized into five categories: retrieval failures, generation failures, tool failures, orchestration failures, and integration failures. For each category, we propose resilience patterns with measured effectiveness from controlled fault injection experiments. Circuit breakers reduce cascade propagation by 89%, output quality gates catch 73% of silent degradation before user impact, and component isolation reduces blast radius by 64%. Systems implementing three or more resilience patterns from our catalog reduce mean-time-to-recovery (MTTR) by 71% compared to unstructured monitoring baselines. We release the incident taxonomy and pattern catalog as a practitioner resource.

Thu 1 OctSoftware EngineeringArtificial IntelligenceDistributed, Parallel, and Cluster Computing
The gist
Big AI systems made of many parts sometimes fail in unexpected ways when these parts interact. The authors studied 150 real failures from different AI projects to figure out common problems and grouped them into five types. They also tested ways to make these systems stronger, like using circuit breakers to stop errors from spreading or quality checks to catch hidden problems early. Systems using several of these fixes recovered much faster after problems than those without them. They share their findings to help others build safer AI systems.
Open → 2610.02503v1

LLM agent action order affects task completion rates significantly

Auditing Action Settlement in LLM Agent Environments: Order, Progress, and Replay

Abstract: Concurrent actions in large language model (LLM) agent environments require arbitration even when each proposal is individually valid. We implement a typed snapshot-settlement contract and audit three distinct properties: order sensitivity, useful progress, and replay consistency. Five settlement policies are tested in 28,800 exhaustive permutation trials and 2,160 scripted multistep episodes. Joint policies are spatially order-invariant conditional on fixed priorities, yet conservative rejection completes only 31.25% of agents in a six-agent doorway task versus 90.28% for random tickets; the paired improvement is 59.03 percentage points (95% bootstrap interval: 50.00-68.06). All policies preserve the tested spatial constraints, and priority arbitration still misses the independent small-instance optimum. A separate full-state journal audit exactly replays 156 checkpoints and rejects 1,332 constructed corruptions with a retained terminal anchor. The evidence concerns execution semantics, not human realism or long-run fairness.

Thu 1 OctArtificial Intelligence
The gist
When multiple AI agents try to act at the same time, their actions might conflict even if each one makes sense alone. The authors studied different ways to decide which action happens first and which get rejected. They tested many scenarios and found that some methods let agents complete tasks much more often. They also developed a system to replay past actions exactly and catch mistakes. This work helps understand how AI agents coordinate their actions, but does not focus on fairness or human-like behavior.
Open → 2610.01138v1

Benchmark evaluates AI agents configuring software deployment environments

FDE-Bench: Evaluating LLM Agents for Deployment Environment Configuration

Abstract: Deployment requires an agent to turn application code into a running system whose services connect, become ready, and remain observable. FDE-Bench evaluates this capability with 136 deployment-configuration tasks spanning Docker images, multi-service Compose stacks, and Kubernetes, in greenfield and diagnose-and-repair modes. Agents submit declarative artifacts that are collected, rebuilt, and redeployed in a pristine environment. Four gated binary check layers measure build, readiness, behavior, and conformance to the deployment specification, using programmatic checks without an LLM judge. A four-arm release gate requires a resolving reference solution and rejects tasks solved by do-nothing, specification-transcription, or generic-stub submissions. The released check annotations expose the link between 2,145 checks and their specifications, including seven documented gaps. Three additional adversarial strategies test shortcuts in the grading signals; none resolves any of the 135 tasks they cover, while a vacuous health probe passes readiness and exposes the need for downstream checks. On the 136-task evaluation grid, seven language models from four providers use the same four-tool scaffold and resolve 52.9-75.0 percent of tasks. The three zero-intelligence floors resolve none and reach a mean Deployment Score of at most 0.44. Readiness is the largest failure stage, accounting for 110 of 313 unresolved episodes. Mean resolution rate is 30.7 percentage points higher on the repair task group than on the disjoint greenfield group, with a positive gap for every model; ten tasks resist all seven. In a 25-task case study, one practicing engineer directing Claude-Sonnet-5 resolves 92 percent against 72 percent for the autonomous baseline. FDE-Bench links deployment success and failure to artifacts that can be inspected and replayed.

Wed 23 SeptSoftware EngineeringArtificial Intelligence
The gist
Setting up software so it runs properly and stays working can be tricky. The authors created a test suite called FDE-Bench that checks how well AI systems can turn code into working software setups using technologies like Docker and Kubernetes. Their tests include building, starting, and checking if the software works as expected, without using AI to judge the results. They found AI can solve many but not all deployment tasks, especially struggling with readiness checks. The benchmark also helps engineers understand and improve deployment processes by linking successes and failures to specific configuration artifacts.
Open → 2609.27571v1

System fixes broken traces across video platform microservices

Backstitch: Restoring Request Causality Across a Production Microservice Fleet

Abstract: A major video platform runs on thousands of microservices, each request propagating a context so downstream work can be traced and governed. At handoffs outside instrumented paths, e.g., custom queues and callbacks, the payload continues but the context does not, and the request still succeeds under existing tests. Such breaks are silent and widespread: 673 of 1,133 services carried at least one. Backstitch, a specialized agentic system, repairs them using the surviving execution as its reference: replay determines whether a suspicious call is request-correlated, source analysis reaches the responsible handoff, a bounded change restores its contract, and the same replay validates the fix. Repairs restore the causal chain without disturbing the work it describes: breaks at 240 of the repaired calls fell from 90.46% to 4.69%, and over 112 days the fleet's break rate more than halved.

Wed 23 SeptDistributed, Parallel, and Cluster Computing
The gist
Many small services handle video requests, passing information along so the whole process is understood. Sometimes the connection breaks silently, causing loss of tracking without failing the request. The authors created Backstitch, a system that finds where these breaks happen and repairs them automatically by analyzing and replaying the service calls. This restores the tracking without changing the actual work, cutting errors in half over months.
Open → 2609.27538v1

Graph structure adds little to microservice error detection accuracy

Does Graph Structure Earn Its Place in Microservice Root-Cause Analysis? A Controlled Study on RCAEval, and What the Benchmark Was Really Measuring

Abstract: Graph neural networks dominate recent work on microservice root-cause analysis, yet recent results question whether the graph contributes. Those results compare whole pipelines, so when a flat model wins one cannot tell whether structure is useless or redundant. We run the comparison they imply on RCAEval: three learned arms with identical features, optimiser, validation split, early-stopping rule and scoring head, in which a single term separates the graph arms. Across two RCAEval benchmarks, two topology sources and four regimes we find no reliable graph-specific effect: in-distribution the graph model leads the conventional flat model by 0.003 Avg@5 (p = 0.844, n = 6 disjoint folds). Auditing the pipeline surfaced two benchmark properties that condition any result on it. RCAEval injects faults into only five services per system while exposing 12 to 70 in telemetry, and the headline metric is Avg@5: a ranker reading no telemetry at all places the true culprit in the top five on 99.7 percent of held-out incidents, scoring Avg@5 0.488. That prior, not the uniform-random 0.137, is the honest in-distribution floor, and it collapses to 0.192 across systems. The second property is a non-uniform column schema that silently zeroes telemetry for most RE1 cases. We reproduce a published baseline, BARO, RCAEval's own reference implementation; on the one system with a clean schema it reaches similar aggregate accuracy to our heuristic, within 0.004, under a different scoring rule. The audit motivated a new model. PSC-GRCA separates a candidate score into a system prior, telemetry evidence and a centred graph residual, and reaches mean Avg@5 0.915 against 0.864 for the flat baseline, while its ablations locate most of the gain in the prior term rather than the graph. We close with a twelve-item checklist for graph-versus-flat ablation studies, distilled from sixty-two recorded defects.

Tue 22 SeptSoftware EngineeringMachine Learning
The gist
Finding the root cause of problems in complex software made of many small parts is hard. The paper studied whether using graph-based models, which consider how parts connect, improves this process compared to flat models that ignore connections. They found that graph models only slightly outperform flat models, and much of the advantage comes from prior knowledge rather than the graph itself. The study also highlights issues in common testing setups that might overstate graph benefits. The authors suggest ways to better evaluate graph use in such tasks.
Open → 2609.27069v1

Agentic AI generates reliable Linux utility programs with human oversight

A Study of the Reliability of Agentic AI-Generated Programs

Abstract: Agentic-AI based software development offers the promise of faster completion of the software, greater programmer efficiency, and more reliable code. The question is how can we verify these claims in an objective way? In this project, we attempted to answer this question based on three practices. First, we applied a typical best-practices agentic AI workflow for software development. Second, our target programs were ten well-known, release-quality human-written Linux utility programs so that we could compare the AI-generated code against a concrete ground truth. Third, we based our measure of reliability on a widely used testing technique, fuzz random testing. For this testing, we used both classic black box, generational testing and more modern coverage guided (gray box, mutational) testing using AFL++. We found that the AI-generated versions of the utility programs were typically as reliable - often more reliable - than the latest human-generated versions of these programs. While the AI-generated versions did have some failures, they were less common than the code from the standard repositories. Interestingly, the AI-generated code was less likely to have failures such as memory errors (such as buffer overflows) but more likely to have hangs such as infinite loops. In addition, we verified that generating robust and reliable software using agentic AI requires careful practice and human supervision. The quality of the code is highly dependent on the prompts and skills used, and how the human directing the process responds. We also demonstrated that using agentic AI workflow for software development (with its prompts and skills) can become a specification of the code that leads to cost-effective sustainability of the software.

Wed 16 SeptSoftware EngineeringArtificial Intelligence
The gist
This study looked at whether programs written by AI can be trusted as much as those written by people. The researchers asked AI tools to create ten well-known Linux utility programs and then tested these AI-made programs with special software tests. They found that the AI-generated code was usually just as reliable or even better than human code, having fewer memory problems but being more prone to getting stuck in loops. However, the quality of AI code depends a lot on how skilled the human guides the AI and what instructions they give. The study shows AI can help make solid software if humans carefully watch and guide the process.
Open → 2609.18298v1