AI summaryⓘ
The authors discuss a problem in reinforcement learning where deciding the best answer at test time is tricky because there is no true reward signal. Instead of just counting how many times answers appear (majority voting), they create a system called CoRE that looks at how answers relate to each other, including similarities and confidence levels. This method forms a network of answers and finds the most agreed-upon group more carefully, improving on simple voting by better recognizing correct minority answers. They tested CoRE on multiple models and datasets, showing it helps learn faster and more accurately without extra cost. Essentially, the authors turn a simple vote into a smarter consensus process for better rewards during learning.
reinforcement learningpseudo-labelmajority votingreplicator dynamicsconsensusreward signaltest-time learningconfidence calibrationdominant setself-supervised learning
Abstract
On unlabeled test data, reinforcement learning lacks a ground-truth reward; test-time RL methods derive one from the model's own roll-outs, rewarding those that match the majority vote over $N$ sampled answers. That vote discards a correct answer whenever it is a minority and scores every majority-matching roll-out identically. We replace it with \emph{CoRE} (Consensus Rewards via Equilibrium): the $N$ roll-outs form a graph whose edges combine answer agreement, reasoning similarity, and generation confidence, and replicator dynamics extract its dominant set, yielding a refined pseudo-label, a graded per-roll-out reward, and a per-question cohesiveness gate. CoRE strictly generalizes voting: majority voting is recovered as a special case; a block-value analysis gives a sharp threshold for when consensus recovers a correct minority against a larger wrong plurality; and confidence calibration provably lowers that threshold multiplicatively. Across seven backbones and five benchmarks (42 model--benchmark cells, three seeds each), \emph{CoRE} improves the untrained base by $+21.7$ points on average versus $+20.4$ for majority-vote TTRL, wins wherever agreement is contestable with margins over the vote of up to $+7.5$ points, and reaches the voting baseline's plateau accuracy in $54$--$70$\% fewer steps. Consensus, not counting: treating the roll-out group as a graph rather than a ballot box turns a brittle vote into a calibrated, graded, self-supervised reward at no extra roll-out cost.