Papers for

crowdsourcing platform managers

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

Partial revelation improves panel vote forecasting accuracy with modest gains

Partial-Revelation Panel Forecasting: Predictive Information Gain, Allocation, and Coarsening-Consistent Operators

Abstract: We study partial-revelation panel forecasting: given $k$ of $n$ annotator votes on the previous dialogue turn, predict the full count distribution of the next turn's panel. This cross-turn predictive information differs from same-item annotation value. Under any nested revelation mechanism, the population decrement satisfies $V(k)-V(k+1)=I(Y;R_{k+1}\mid X_k)$; a fitted model also contributes adjacent excess-risk differences. Coarsened-completely-at-random (CCAR) masking supplies the exchangeable random-rater interpretation used in our operators. We define coarsening-consistent panel operators with a closed-form Dirichlet-multinomial realization, learned completion, and rater-count transfer. Against per-level lookup, cold-start count negative log-likelihood differences are -0.422/-0.166/-0.014 nats on IEMOCAP/M3ED/EmoWOZ. A held-out rater experiment measures same-item value: first and second additional votes reduce NLL by 0.583/0.221, 0.578/0.184, and 0.180/0.069 nats, respectively. Cross-turn and same-item gains are not interchangeable, and CCAR masks are controlled interventions rather than observed arrival processes. Consistency is auditable but does not guarantee predictive improvement; per-level marginalization can outperform a shared table. Target-free allocation beats uniform only where row-level signal is predictable: at budget one it recovers 83--93% of retrospective feature class headroom on IEMOCAP/M3ED but does not improve EmoWOZ. Gains are modest and corpus-dependent, with several small or null partial-revelation differences. Code and aggregate result artifacts accompany the preprint.

Fri 16 OctMachine Learning
The gist
This paper looks at predicting the full results of a group vote using only part of the votes counted so far. The authors develop math tools to better guess the final tally from partial information and test these on emotional dialogue datasets. They find that knowing some votes helps predict the rest, but the improvements depend on the dataset and are often small. Their methods also help decide how to allocate limited voting resources for better forecasts. The work includes code and results for others to build on.
Open → 2610.22831v1