Language models struggle to predict shifting research focus in ai fields
RAP: Research Attention Prediction Reveals Target-Conditioned Evidence Acquisition Biases
Artificial IntelligenceComputation and Language
Summary
It is hard to tell if large language models can predict where research will focus next because research ideas don’t have clear, verifiable outcomes. The authors created a way to test this prediction by tracking 278 AI areas and checking how well models guess which topics will be popular in six months. Their tests show that simple past-trend methods often predict better than complex model strategies. They also found that models have trouble updating their predictions based on new information about recent research activity. Training a model on actual past outcomes improved predictions somewhat but challenges remain.
What this means in practice
- •For ai product managers: Improve planning by knowing the limits of language models in forecasting research trends over the short term.
- •For data science teams: Design evaluation benchmarks that measure how models update predictions when new research appears.
Authors
Yingqian Wu, Jingcong Liang, Siyuan Wang, Zhenfei Yin, Philip Torr, Junchi Yu, Zhongyu Wei
Abstract
Large language models (LLMs) increasingly act as research agents, yet their ability to track shifts in research attention is difficult to evaluate because reviews and research ideas lack uniquely verifiable outcomes. We introduce Research Attention Prediction (RAP), a rolling benchmark covering 278 AI/ML fields and 1,390 episodes. At each cut-off, an LLM agent searches a temporally restricted arXiv corpus and predicts the next six months' paper shares across eight frozen research directions. Search generally helps, but all four diagnostic models perform worse than an exact-count exponentially weighted moving average (EWMA) baseline in compositional accuracy. We identify two linked bottlenecks. Under cumulative-history access, State carry-forward outperforms direct Forecast for all four diagnostic models; frozen-evidence replay links a shared component of this reversal to Forecast-oriented policies retrieving a smaller share of recent evidence. Even with exact historical activity, future-specific updating remains limited, with only GPT-5.5 plus reopened Search slightly surpassing EWMA. Fine-tuning on realised outcomes improves Qwen3-4B's forecast Spearman correlation by 0.105 on held-out fields at later origins, with gains also on change-rich episodes.