AI summaryⓘ
The authors study systems where multiple evaluators vote yes or no to accept a solution for a language task, and these votes are combined to make a decision. They focus on how consistent these decisions are when compared to a defined reference group of evaluators, rather than on whether the solution is objectively true. They analyze different scenarios including small fixed groups, large populations, and limits as group size grows, providing mathematical methods to estimate disagreement rates and confidence bounds. Their approach carefully handles dependencies between evaluators and uses exact statistical techniques. They also tested their methods with simulations to verify accuracy and understand limitations.
Natural-language evaluationAggregate disambiguation systemsProtocol reproducibilityBinary votingStatistical confidence boundsEvaluator populationsSampling methodsBinomial intervalsDisagreement probabilitySimulations
Authors
José María Lago, Albert Castellana, Edgars Nemše
Abstract
Natural-language tasks can elicit different verdicts from protocol-following evaluators that receive the same declared information. We study aggregate disambiguation systems (ADSs). Given a task and a candidate solution, each evaluator casts a binary vote on whether the solution should be accepted, and the system aggregates the votes of a finite panel. The target is protocol reproducibility relative to an explicitly declared evaluator reference, not semantic truth. We separate fixed finite censuses, probabilistic evaluator populations, and growing-census limits, since their endpoint laws and guarantees are not interchangeable. In the population setting, we use finite samples to estimate how often a finite panel reaches the same decision as the declared evaluator population. We provide a lower confidence bound on the fraction of candidate solutions for which the disagreement probability is at most a chosen tolerance. The calculation accounts separately for sampling candidate solutions and sampling evaluators. The construction permits arbitrary dependence among columns induced by shared evaluator rows and uses exact binomial intervals at the evaluator layer and an exact one-sided binomial inversion at the generator layer. Simulations check the implementation against known population coverages and expose power limitations.