Harnessing execution feedback to improve program generation performance
HERO: Harnessing Evidence for Test-time Refinement and Prompt Optimization in Program Generation
Computation and Language
Summary
Generating correct code from text instructions is hard, and prompts used to guide code generation can sometimes cause errors or lose good behavior. The authors introduce a method called HERO that learns from running generated code to see what works and what doesn't. It updates prompts based on this feedback and even refines the code generation at test time using execution results. This approach improves the accuracy of generating SQL, Cypher, and Python code compared to previous methods.
What this means in practice
- •For software developers: Improve automated code generation tools by using execution feedback to refine prompts and generate more accurate code snippets.
- •For database administrators: Enhance natural language to database query translation by learning from verified execution outcomes to reduce query errors and increase reliability.
Authors
Mingyu Lu, Timothy Ossowski, Tristan Naumann, Hoifung Poon, Peniel Argaw
Abstract
Harness optimization improves LLMs by refining reusable prompts and scaffolding, yet requires further adaptation for program generation tasks such as text-to-SQL and text-to-Python. Existing methods often rely on LLM reflectors to propose harness updates, which can fix observed failures but also disrupt previously correct behavior. Text-to-code tasks offer a natural way to mitigate this issue: we can execute generated outputs, allowing us to evaluate and verify the effects of proposed updates directly. Building on this signal, we introduce \textsc{HERO} (\textbf{H}arnessing \textbf{E}vidence for \textbf{R}efinement and \textbf{O}ptimization), which combines offline and test-time harness optimization. At the first stage, \method{} learns reusable prompt updates from execution-verified successes and failures and validates candidate revisions before adoption. At test time, we treat the generated code as an instance-specific harness and refine it using execution feedback. Across four SQL, Cypher, and Python benchmarks, \method{} achieves the best offline performance, improving pass@1 by \(7.6\) points over the initial prompt, while test-time refinement adds another \(2.0\) points and improves execution efficiency.