Tracking detailed changes improves automatic prompt tuning outcomes
Beyond Aggregate Accuracy: Item-Level Transitions in Prompt Optimization
Machine LearningArtificial IntelligenceComputation and Language
Summary
Improving prompts for AI models often focuses on the overall number of right answers but misses which questions get better or worse. The authors created a tool called RFDPO that tracks when individual answers improve, worsen, or change a lot during tuning. Using this detailed tracking helped reduce unnecessary flips in answers while keeping overall improvement steady. They tested this on several tasks using GPT models and found that looking beyond just overall accuracy gives a clearer picture of prompt optimization.
What this means in practice
- •For ai system developers: Use item-level feedback tracking to improve prompt tuning efficiency and stability in large language models.
- •For ai application engineers: Assess prompts by monitoring individual question-level changes, not just overall accuracy, to better maintain model reliability.
Authors
Chinmoy Mitra, Tarek Mahmud
Abstract
Automatic prompt optimization (APO) uses feedback from model responses to improve prompts automatically. However, its effectiveness is usually measured through overall scores, which do not reveal which questions become correct, which previously correct answers become wrong, or whether improvements extend to unseen questions. We introduce Reflective Feedback-Driven Prompt Optimization (RFDPO), a tool that tracks recoveries (incorrect to correct), regressions (correct to incorrect), and churn (all changed outcomes). RFDPO uses this feedback to guide subsequent revisions and records prompt changes throughout optimization. We evaluate RFDPO across five benchmarks covering law, knowledge, instruction following, mathematics, and privacy, using GPT-5 as the optimizer and GPT-4o-mini and GPT-4.1 Mini as solvers. Our results show that overall accuracy can hide substantial changes: in five of nine configurations, churn was at least five times the absolute net change, and a single-item net change concealed up to 17 changed outcomes. Average validation gains were 7.9 percentage points, compared with 5.2 points on held-out tests. In a controlled ablation on LegalBench-Hearsay, item-level feedback reduced churn by 40% ($p=0.024$) while maintaining similar net gains. These findings motivate reporting recovery, regression, churn, and performance on unseen data alongside overall accuracy.