Evaluation of language models by success after repeated feedback attempts
Seq@K: Evaluating Language Models Under Sequential Feedback
Artificial Intelligence
Summary
Many language models try to solve problems by giving one answer, but in real life, people usually improve answers by getting feedback and trying again. The authors introduce Seq@K, a way to test if a model can solve a problem within K tries while learning from feedback after each mistake. They found that models rank differently under this test compared to old methods, helping to identify how well models really use feedback to improve. This method also helps understand how models change their answers over attempts and shows that the type of feedback matters for improving models.
What this means in practice
- •For ai product teams: Select language models that improve answers efficiently through user feedback in interactive AI applications.
- •For conversational interface developers: Design and test dialogue systems that adapt better to user corrections and iteratively improve responses.
Authors
Thomson Yen, Gilbert Yang, Naimeng Ye, Minghui Chen, Dong Xia, Tianyi Shen, Yechan Jeon, Yaqin Chen, Daksh Mittal, Fangyu Wu, Hongseok Namkoong
Abstract
Language-model agents differ fundamentally from prior intelligent systems in their ability to learn from natural language in context. This capability enables agents to iteratively refine their answers in response to feedback, a prevalent mode of operation in modern agentic workflows. However, the community primarily ranks models by metrics such as Pass@1 or Pass@K, which cannot capture how they utilize feedback on prior attempts. We introduce Seq@K, a natural evaluation protocol that measures whether a model can solve a problem within K sequential attempts, provided feedback after each previous failure. Unlike multi-turn agentic benchmarks that examine long-horizon planning, Seq@K isolates and formalizes the core capability of refinement from feedback. We evaluate Seq@K on a suite of 7 benchmarks and 16 frontier models, and show that Seq@K reveals a materially different model-selection landscape: under Seq@K, (1) model rankings often differ, (2) the Pareto frontiers admit more efficient configurations and (3) the gains from choosing the optimal model configuration can widen or shrink relative to those of Pass@K. We further show that analyzing model trajectories under Seq@K helps characterize iterative refinement behavior, such as the rate at which models address issues raised in feedback, or how rigidly they hold onto their own answers. Finally, we study how the nature of feedback affects models' iterative refinement, showing that practitioners must carefully design and benchmark the feedback loops of iterative agentic systems. Together, these results establish how Seq@K offers a more faithful metric for intelligent systems that typically operate in such feedback loop, and the necessity to consider Seq@K in evaluating models for these systems.