Papers for

software integration engineers

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

LLM method guarantees valid imports in generated tests without errors

No More Import Errors: Constrained Decoding for Guaranteed Valid Imports in LLM-Generated Tests

Abstract: One of the most common failure reasons of LLM-based test generation pipelines is import errors. Grammar constrained decoding can modify the probabilities of the LLM's final layer during token generation to guarantee valid imports. However, we find that it can lead to degenerate solutions, i.e., repetitive valid imports (import streaks) that exhaust the token budget before a test is generated. We propose Constrained Beam Decoding with Lookup (CBDL), a novel method that guarantees import validity while addressing import streaks by avoiding premature commitment to tokens that lead to repetitive imports. Because it operates at the decoding stage, CBDL can be integrated into existing LLM-based test generation pipelines. We implement CBDL for Python and Java and evaluate it on two test generation tasks, showing that generated tests have 100% valid imports, outperforming both static (RAG and LIBRO) and dynamic (Agentic Repair) import-validity approaches that leave 45%, 10%, and 43% of import errors unresolved. CBDL converts on average 14% of initial import errors into functionally correct tests, increasing overall test correctness by 1-4 percentage points over unconstrained generation. Even when import validity does not directly lead to a correct test, it allows agentic repair pipelines to allocate the repair budget to remaining errors, increasing correct tests by 2-7 percentage points for Python and 1-6 for Java. Finally, we show that a hybrid pipeline, where a small local model equipped with CBDL generates imports while the frontier GPT-5.2 model generates the test body, outperforms generating both with GPT-5.2 alone by 1-3 percentage points. Overall, CBDL removes a major bottleneck in LLM-based test generation: by guaranteeing import validity with a small overhead (16-21 seconds per test), it improves test correctness and enables cost-effective hybrid generation.

Fri 30 OctSoftware Engineering
The gist
Generating tests using large language models often fails because the code imports are invalid, causing errors. The authors propose a new way called Constrained Beam Decoding with Lookup (CBDL) that prevents these import errors while avoiding getting stuck in repetitive import patterns. This method can be added to existing test generation workflows and works for languages like Python and Java. Their approach fixes all import errors and improves the correctness of generated tests by reallocating effort to other issues. It also enables cheaper hybrid setups that split import and test generation between models.
Open → 2610.41878v1

Llm agents improve firmware coding through iterative testing and repair

Closed-loop evaluation of LLM agents for embedded software development

Abstract: Large language models (LLMs) are increasingly deployed as coding agents that edit files, run builds and tests, inspect execution results, and repair software iteratively. Embedded firmware is a demanding target because correctness depends on closed-loop behavior under sensing, timing, and safety constraints, not only on static source quality. Yet embedded-agent evaluation remains limited and often emphasizes one-shot synthesis or offline correctness. We present a benchmark for closed-loop evaluation of embedded coding agents. Each task provides a plain-text engineering description, constrained workspace, and visible build-and-runtime surface. The agent must translate requirements into implementation and self-verification steps, then iterate until the required device behavior is achieved. The suite contains five embedded-control tasks and four feedback scenarios: one-shot generation, realistic self-verification, CI-style red/green feedback, and oracle-style detailed feedback. The implementation targets simulated ESP32 firmware for reproducibility. We evaluate seven GPT-family and Qwen-family configurations across five tasks and four scenarios, with three repetitions per condition for 420 runs. gpt-5.4 has the highest pass rate among evaluated configurations but does not saturate the benchmark; qwen3.5-27B is the strongest observed local model; and smaller local models degrade sharply in pass rate and search efficiency. These results suggest that capable local embedded coding agents are emerging.

Thu 8 OctSoftware EngineeringArtificial IntelligenceHardware Architecture
The gist
Writing software for devices like microcontrollers is tough because the code must work correctly in real-time and under specific safety conditions. This paper introduces a new way to test how well large language models (LLMs) can write, test, and fix embedded software by simulating this real-world process repeatedly. The authors evaluated several LLMs on tasks involving simulated firmware for a common microcontroller and found that more advanced models performed better but still have room to improve. This helps show how close these AI tools are to being reliable assistants for embedded software development.
Open → 2610.11447v1