Benchmark measures summarization of IT incident logs from synthetic data
SLS-Bench: A Benchmark for Incident Log Summarization with Synthetic Observability Data
Artificial Intelligence
Summary
As computer systems become more complex, it’s important for automated tools to turn long sets of error logs into easy-to-understand reports. The authors created SLS-Bench, a test set of 90 examples where synthetic log data is paired with summary reports based on real incidents. They developed a way to generate realistic fake log data that matches real-world patterns using models and verification steps. Testing different large language models shows some can create helpful summaries, but with delays and high costs, while smaller models struggle more. This benchmark helps measure and improve tools that analyze large IT incident logs automatically.
What this means in practice
- •For it operations teams: Evaluate and improve automated incident log summarization tools using realistic synthetic data modeled on real incidents.
- •For cloud service engineers: Test and scale log analysis agents for large observability data streams without needing costly real-world data collection.
Authors
Valerii Iakovlev, Markus Spanring, Martin Flechl
Abstract
As IT systems grow in complexity, automated agent-driven incident response becomes increasingly critical. A key capability for these agents is summarizing large log streams into human-readable reports. Further progress requires meaningfully evaluating this function, yet existing benchmarks leave end-to-end summarization from large log streams underexplored due to the prohibitive cost and complexity of collecting real-world data. We introduce SLS-Bench, a benchmark for incident log summarization featuring 90 tasks grounded in real-world incident reports, each with a synthetic log stream and a reference summary. To construct SLS-Bench, we developed a data generation pipeline that converts incident reports into structured incident models, which guide the generation of synthetic logs and reference summaries. This pipeline combines narrative grounding, explicit incident modeling, generator-verifier loops, and statistical comparisons with real-world log streams to produce realistic and controllable observability data. Our evaluation of 14 LLMs on SLS-Bench shows that while frontier models can produce useful summaries, they do so at substantial cost and latency, and open-weight models lag significantly. SLS-Bench provides a framework for systematic evaluation of log analysis agents and a scalable blueprint for future synthetic observability benchmarks.