Execution trace debugging improves fixing of assertion failures in ai code

On the Limits of Execution Traces for Repairing Assertion Failures in LLM-Generated Code

Software EngineeringArtificial Intelligence

Summary

Fixing errors in code generated by large language models (LLMs) is hard, especially when the errors are about failing checks inside the program (assertions). The authors found that simply having the error message doesn’t give enough clues to fix the code. They proposed adding detailed run-time information about the program’s variables at the point of failure, called Breakpoint Debugging, which helps the model repair the code better in many cases. However, this approach isn’t always better than just trying new fixes from scratch, especially when the failing tests don’t reveal enough about the problem.

What this means in practice

  • For software developers: Use breakpoint debugging traces to improve automatic repair of AI-generated code with assertion failures.
  • For ai tool builders: Incorporate runtime variable information into iterative repair loops to enhance debugging and code fixing in AI code generation tools.

Authors

Muntasir Adnan, Carlos C. N. Kuhn

Abstract

Iterative repair returns a failed program's error to a large language model (LLM) and asks it to fix it. After iterative repair, the remaining failures are mostly assertion failures: with four Qwen2.5 models, they make up 79% of residual failures on BigCodeBench, all on HumanEval and 93% on LiveCodeBench. We argue, through an information view of the feedback, that an assertion failure says little about where the fault lies. We propose Breakpoint Debugging (BPD), which returns the failed program annotated inline with its variables' runtime values. We take the tasks that iterative repair failed to fix with five repair attempts and compare two ways of spending five further generations on them: BPD, and resampling. On BigCodeBench, BPD repairs 26.7% of these tasks against 22.4%, and its advantage grows with model size. On HumanEval, BPD outperforms resampling but only slightly, and on LiveCodeBench, fresh sampling repairs more (6.5% against 4.9%). Furthermore, BPD reduces unchanged-code resubmissions from 28% to 8% on BigCodeBench. Using only the first failing test's trace, it outperforms tracing all failing tests while using fewer tokens than both all-trace BPD and plain iterative repair. On LiveCodeBench, BPD gives no gain even on the public tests, so a visible failing input is not enough for a trace to help. These results match the information view: a trace helps only when the test exposes a failing input, and the fault is an implementation slip; otherwise, a repair loop should resample rather than trace.