Aggregating weaker language models improves event forecasting accuracy
Two Heads Are Better Than One: Aggregating Weaker LLMs for Better Forecasts
Machine Learning
Summary
Forecasting future events using language models can be expensive when relying on the strongest models. The authors show that combining multiple weaker language models can match or even beat the best single model’s predictions. They tested this approach on many groups of models and found consistent improvements using a method that learns how to weight each weaker model’s forecasts. Adding more models beyond pairs doesn’t always help, but well-chosen pairs still provide reliable better forecasts.
What this means in practice
- •For data science teams: Combine forecasts from multiple less powerful language models to improve predictions without costly access to the top model.
- •For financial analysts: Use aggregated weaker language models to generate accurate event forecasts when premium models are unavailable or too expensive.
Authors
Cheng Peng, Ruixi Luo, Zhi Chen, Wei Tang
Abstract
Large language models (LLMs) are increasingly used to forecast real-world events, but access to the strongest individual forecaster may be costly or otherwise constrained. We study weak-to-strong forecast aggregation: can individually weaker LLM forecasters be aggregated to outperform a stronger forecaster? Using ForecastBench (Karger et al., 2025), we evaluate 70 LLM forecasters across 16 comparison groups, each with more than 1,000 shared subquestions, yielding 1,121 weaker-model pairs. Within each group, we identify the strongest individual by test Brier score and evaluate aggregates composed exclusively of weaker forecasters, with aggregation weights learned on separate training data. We find substantial evidence of weak-to-strong improvement. Learned linear pooling identifies a weaker pair that matches or outperforms the strongest individual in 11 of 16 groups and comes within 5% of its Brier score in all 16 groups. We also find that these improvements do not rely on having a near-best constituent and are generally accompanied by good calibration. Additional analyses show that adding more models does not consistently improve performance, and competitive weaker-model aggregates also remain available under practical constraints.