ParEvalLayer: When Partial LLM-Agent Evaluations Support a Decision
2026-08-03 • Artificial Intelligence
Artificial Intelligence
AI summaryⓘ
The authors explain that when evaluating large language model (LLM) agents, stopping early and reporting partial scores can be misleading because early tasks might not represent the full challenge fairly. They propose ParEvalLayer, a method that looks at partial results and decides if enough information exists to declare one agent better or if more testing is needed. By testing this method on existing benchmark data, they found that for some tasks, decisions can be confidently made after seeing only 15% to 25% of the results, but others need more data. This shows that partial scores should always come with clear decision rules and notes about how many comparisons are still undecided.
LLM-agent evaluationpartial scoringbenchmark runscomparison policydecision layeragent rankingevaluation stopping criteriatask outcomesbenchmark completenessParEvalLayer
Authors
Wei-Jung Huang, Bonan Shen
Abstract
LLM-agent evaluations often produce task outcomes long before the full benchmark run is complete. A partial score is tempting to report, but it does not show whether the observed tasks support the same conclusion as the completed evaluation. Early tasks can omit important parts of a benchmark, running cheaper tasks first can distort the observed sample, and a rule that decides only easy pairs can appear accurate while leaving many comparisons unresolved. We introduce ParEvalLayer, a decision layer that reads paired outcomes for two agent systems and a comparison policy chosen in advance. For each partial run, it records whether the tested agent system is better by the required amount, is not better by that amount, needs more evidence, or should abstain. We evaluate ParEvalLayer by replaying completed public benchmark data as if each evaluation had stopped earlier. At each point, ParEvalLayer applies the policy using only the outcomes observed so far; if it reaches one of the two comparison judgments, we check whether that judgment matches the completed data for the same system pair. With the main comparison rule, three of the public benchmarks reach the same decision as the completed evaluation after observing only 15% to 25% of task outcomes. Other benchmarks require more task outcomes. This variation shows why a partial score alone is not enough: reports should also state the decision rule and how many comparisons remain without a decision.