LLM agents improve resilience in high performance computing applications
LLM Agents as Resilience Engineers for Scientific Applications
Distributed, Parallel, and Cluster Computing
Summary
High-performance computing (HPC) applications need a way to save their work regularly so they can recover quickly from failures. This saving and recovery process is hard to implement because programmers must carefully identify and manage the important parts of the program's state. The authors tested whether advanced language models (LLMs) could automate this task. They found that LLMs can successfully generate checkpoint and restart code in many cases within about an hour, especially when the program’s important data is in clear and accessible structures. However, they struggle when the important data is scattered or complex.
What this means in practice
- •For hpc application developers: Automate checkpoint/restart code generation for HPC programs with accessible state for faster and reliable failure recovery.
- •For software reliability engineers: Use LLM-driven pipelines to create recovery mechanisms in scientific computing systems, reducing manual coding errors and effort.
Authors
Hai Duc Nguyen, Tekin Bicer, Kyle Chard, Ian Foster, Bogdan Nicolae
Abstract
Efficient checkpoint/restart support is essential for resilient HPC scientific applications, but implementing it requires substantial expertise: developers must identify recoverable state, choose globally consistent checkpoint points, and preserve application invariants during restart. We study whether frontier LLM coding agents can automate this process. We build a benchmark suite of 16 MPI applications spanning diverse domains, code sizes, and critical-state structures, and evaluate them with a no-human-in-the-loop generate--validate--revise pipeline for checkpoint/restart synthesis. Across the benchmark, the pipeline produces 41 working resilient implementations. Our results show that agent-driven resilience engineering is practical when critical state is visible or accessible through coherent abstractions: successful runs finish in under one hour on average, consume about 15M tokens, and produce implementations with negligible failure-free overhead and recovery efficiency comparable to human-written code. However, modularized and fragmented state remains a major limitation, with some failed attempts consuming over 100M tokens and 300 minutes without producing a working implementation.