Llm agents improve firmware coding through iterative testing and repair

Closed-loop evaluation of LLM agents for embedded software development

Software EngineeringArtificial IntelligenceHardware ArchitectureMachine Learning

Summary

Writing software for devices like microcontrollers is tough because the code must work correctly in real-time and under specific safety conditions. This paper introduces a new way to test how well large language models (LLMs) can write, test, and fix embedded software by simulating this real-world process repeatedly. The authors evaluated several LLMs on tasks involving simulated firmware for a common microcontroller and found that more advanced models performed better but still have room to improve. This helps show how close these AI tools are to being reliable assistants for embedded software development.

What this means in practice

  • For embedded software teams: Use LLM-based coding agents that iteratively test and fix firmware to improve development speed and correctness under realistic device constraints.
  • For software integration engineers: Evaluate LLMs for their ability to run builds and tests in an automated loop to support continuous integration processes on embedded systems.

Authors

Jorge García-Carrasco, Sergio García-Carrasco, Alejandro Maté, Juan Trujillo

Abstract

Large language models (LLMs) are increasingly deployed as coding agents that edit files, run builds and tests, inspect execution results, and repair software iteratively. Embedded firmware is a demanding target because correctness depends on closed-loop behavior under sensing, timing, and safety constraints, not only on static source quality. Yet embedded-agent evaluation remains limited and often emphasizes one-shot synthesis or offline correctness. We present a benchmark for closed-loop evaluation of embedded coding agents. Each task provides a plain-text engineering description, constrained workspace, and visible build-and-runtime surface. The agent must translate requirements into implementation and self-verification steps, then iterate until the required device behavior is achieved. The suite contains five embedded-control tasks and four feedback scenarios: one-shot generation, realistic self-verification, CI-style red/green feedback, and oracle-style detailed feedback. The implementation targets simulated ESP32 firmware for reproducibility. We evaluate seven GPT-family and Qwen-family configurations across five tasks and four scenarios, with three repetitions per condition for 420 runs. gpt-5.4 has the highest pass rate among evaluated configurations but does not saturate the benchmark; qwen3.5-27B is the strongest observed local model; and smaller local models degrade sharply in pass rate and search efficiency. These results suggest that capable local embedded coding agents are emerging.