Self-improving system halves time to solve repeated software faults

A Self-Defining Operator for Distributed Applications

Distributed, Parallel, and Cluster ComputingArtificial Intelligence

Summary

Fixing problems in large distributed applications can be slow because computers don’t remember past fixes. The authors created a system called self-defining operator (SDO) that learns from each problem it solves by saving step-by-step instructions and patterns to recognize similar issues. This way, SDO gets faster over time and only uses costly language queries when needed. When tested on multiple software projects, SDO cut the time to fix repeated problems nearly in half while keeping costs low.

What this means in practice

  • For software reliability teams: Automatically accelerates troubleshooting by reusing past fixes to diagnose and solve recurring faults in distributed systems.
  • For cloud operations engineers: Reduces operational costs by limiting expensive language model usage to incidents resembling prior known failures in cloud-based microservices.
  • For application performance monitoring vendors: Enable products that provide intelligent, learned troubleshooting playbooks to clients managing distributed applications.$Commercial implications: This system supports commercial tools offering adaptive diagnostic automation for distributed software, improving service offerings and client value.

Authors

Shihang Li, Thomas Anderson, Ratul Mahajan, Simon Peter

Abstract

To operate distributed applications with high availability, operators must rapidly diagnose and mitigate any incident. LLM-based agents can help, but unlike experienced human operators they do not improve as they handle more incidents. Each investigation starts from scratch, re-deriving likely causes and re-testing hypotheses even when a similar fault was resolved in the past. We present a self-defining operator (SDO) that improves with time by updating its operational behavior and memory. When SDO resolves an incident, it distills that experience into playbooks, which describe reusable diagnostic instructions for future incidents with similar faults, and detectors, scripts that take application telemetry as input and output relevant playbook(s). SDO continuously runs the detectors and invokes LLMs only when one fires, so its token cost scales with incident count rather than with operational time. Across 11 open-source microservice applications, SDO cuts resolution time by 52% on recurring faults while keeping per-incident cost low. SDO is open source at https://github.com/uw-syfi/sdo.