Benchmark evaluates AI agents configuring software deployment environments
FDE-Bench: Evaluating LLM Agents for Deployment Environment Configuration
Software EngineeringArtificial Intelligence
Summary
Setting up software so it runs properly and stays working can be tricky. The authors created a test suite called FDE-Bench that checks how well AI systems can turn code into working software setups using technologies like Docker and Kubernetes. Their tests include building, starting, and checking if the software works as expected, without using AI to judge the results. They found AI can solve many but not all deployment tasks, especially struggling with readiness checks. The benchmark also helps engineers understand and improve deployment processes by linking successes and failures to specific configuration artifacts.
What this means in practice
- •For devops teams: Assess AI agents' capabilities to automate and verify deployment configurations across diverse container and orchestration environments.
- •For software reliability engineers: Diagnose and improve automated deployment processes by inspecting artifacts linked to deployment successes and failures within the benchmark.
Authors
Weihang Ding, Junfei Zhan, Yueting Li, Qirong Guo
Abstract
Deployment requires an agent to turn application code into a running system whose services connect, become ready, and remain observable. FDE-Bench evaluates this capability with 136 deployment-configuration tasks spanning Docker images, multi-service Compose stacks, and Kubernetes, in greenfield and diagnose-and-repair modes. Agents submit declarative artifacts that are collected, rebuilt, and redeployed in a pristine environment. Four gated binary check layers measure build, readiness, behavior, and conformance to the deployment specification, using programmatic checks without an LLM judge. A four-arm release gate requires a resolving reference solution and rejects tasks solved by do-nothing, specification-transcription, or generic-stub submissions. The released check annotations expose the link between 2,145 checks and their specifications, including seven documented gaps. Three additional adversarial strategies test shortcuts in the grading signals; none resolves any of the 135 tasks they cover, while a vacuous health probe passes readiness and exposes the need for downstream checks. On the 136-task evaluation grid, seven language models from four providers use the same four-tool scaffold and resolve 52.9-75.0 percent of tasks. The three zero-intelligence floors resolve none and reach a mean Deployment Score of at most 0.44. Readiness is the largest failure stage, accounting for 110 of 313 unresolved episodes. Mean resolution rate is 30.7 percentage points higher on the repair task group than on the disjoint greenfield group, with a positive gap for every model; ten tasks resist all seven. In a 25-task case study, one practicing engineer directing Claude-Sonnet-5 resolves 92 percent against 72 percent for the autonomous baseline. FDE-Bench links deployment success and failure to artifacts that can be inspected and replayed.