Batching improves speed and results in reflective text optimization
Batching the Reflective Optimization Loop: Parallel Proposals Make GEPA Faster and Better
Artificial IntelligenceComputation and LanguageMachine Learning
Summary
When computer programs try to improve their own text generation step by step, the process can be slow because it does one suggestion at a time. The authors show that making many suggestions all at once and checking them in parallel is faster and leads to better results. They updated a tool called GEPA to do this batching, cutting the time it takes by half or more and improving accuracy on test tasks. They also explain why this approach saves time and resources.
What this means in practice
- •For optimization engineers: Reduce time and cost when tuning AI models by running multiple candidate solutions in parallel during reflective optimization.
- •For natural language processing teams: Improve model evaluation throughput and accuracy when developing text-based AI systems using batch candidate proposals.
Authors
Jialin Zhang, Lakshya A Agrawal, Shangyin Tan, Donghyun Lee, Lutfi Eren Erdogan, Dan Klein, Koushik Sen, Alex Dimakis, Matei Zaharia
Abstract
Reflective text optimization makes agentic systems better, but the optimization loop is itself a resource-hungry workload whose sequential structure leaves parallel evaluation capacity idle. We extend the open-source GEPA optimizer to propose and evaluate a batch of candidates on each optimization step instead of one candidate at a time. In our sweep on two tasks, most batched runs finished in half the wall-clock time or less, and the fastest in about a quarter to a third. Batched settings also achieved higher held-out test scores, from 68.9% to 72.1% on LiveBench-Math (with $2\times2$) and from 49.0% to 60.0% on HoVer (with $8\times1$). A simple runtime model explains the measured speedups under a finite worker pool, and batched settings also spend their metric-call and dollar budgets more efficiently. We frame the axes along which a reflective optimization step can scale in parallel, a first look at allocating a fixed optimization budget across them.