Prompt tuning produces no clear gains on top reasoning AI models
Compile as Hypothesis: A Headroom Typology for Prompt Optimization on Real Reasoning Models
Artificial Intelligence
Summary
This paper studies whether improving prompts given to advanced AI reasoning models consistently makes them perform better. The authors found that tweaking prompts often does not lead to meaningful accuracy improvements for strong models, although it helps avoid certain failures. They show that prompt optimization cannot fix knowledge gaps or input size limits, but can correct formatting mistakes and instruction clarity. The paper also uncovers hidden technical issues in commercial AI services and provides tools to verify evaluation reliability.
What this means in practice
- •For ai system developers: Detect and prevent failure modes related to uncontrolled prompt modifications in advanced reasoning models during integration.
- •For api service operators: Implement automated tools for per-call auditing to ensure consistent and accurate evaluation of AI model responses during outages or backend changes.
Authors
Zhengshuai Yang, Changyun Li
Abstract
Automated prompt compilers search for or rewrite prompts under the implicit assumption that optimization consistently improves downstream task performance. We evaluate this assumption on frontier reasoning models through rigorous paired, same-day measurements. Across four optimizer families (genome search, profile-guided mutation, GEPA, and ESPO) and eight tasks, structured search yields no statistically significant accuracy gains on strong reasoners. However, a discrete structural genome provides essential architectural stability, preventing catastrophic bimodal seed-collapse (where unconstrained search collapses toward the judge floor in 3/5 runs). Through 190-problem evaluations on knowledge-intensive benchmarks, we establish a three-dimensional headroom typology: prompt optimization cannot compensate for intrinsic knowledge deficits or physical token truncation, but successfully resolves protocol and formatting errors. On gpt-6, while free-text reflective search (GEPA) yields a +.14 gain on a task where structured search remains flat, qualitative ablation reveals that ~.035 stems from judge-rubric reconstruction while ~.105 reflects genuine instruction refinement. Our pre-registered AutoAPC-Select deployment gate matches the oracle policy in 5/7 groups and rescues seed-collapse arms with zero regret. Finally, we expose silent backend-mixing in commercial APIs during outage windows and provide automated per-call audit verification tools to ensure empirical evaluation fidelity.