AI language models struggle to match human research idea choices
IdeaTaste: Do LLMs Share the Research Taste of Human Researchers?
Computation and Language
Summary
This work studies how well large language models (LLMs) pick research ideas like human scientists do. The authors find that AI proposals match only about a quarter of the citations and fewer than one-fifth of the ideas found in real future research papers. While AIs can spot broad topics and methods, they have trouble turning those into concrete research plans that humans pursue. They also observe that AI-generated ideas not shared by any human paper tend to be less promising directions. This benchmark helps evaluate and guide AI’s ability to choose worthwhile research topics.
What this means in practice
- •For ai development teams: Improve AI research suggestion tools by benchmarking them against future actual research to better guide AI towards human-aligned research directions.
- •For science policy planners: Use AI-generated research idea evaluations to support decisions on funding or prioritizing emerging research fields where human taste is hard to predict.
Authors
Shanshan Zhong, Chen Xu, Jingjie Ning, Zhengzhong Liu, Aviral Kumar, Chenyan Xiong
Abstract
Research taste, the judgment of what to read, which problems to pursue, and which methods to try, increasingly shapes the work of AI scientists, yet it remains far less understood than their ability to execute research. We introduce IdeaTaste, a benchmark that scores an LLM by how closely the research proposals it writes from a literature snapshot frozen at 2025 match the work the same communities published in 2026. Our experiments span six contemporary language models and four research areas from established to emerging fields, covering more than 90{,}000 proposals in total. LLM agents fall well short of human communities in both the literature they cite and the ideas they state, reaching about 26\% of the human level on citation matching and only 18\% on idea matching. Their populations can cite as diversely as a community and take up its tasks and methods, yet rarely arrive at the concrete approaches and research plans it pursues. Neither richer field context nor larger populations substantially close this gap. In blinded expert review, AI proposals trail the 2026 papers least in presentation and most in expected success, echoing their gap in concrete approaches, and those whose ideas no human paper shares fare worst, suggesting weaker directions rather than overlooked opportunities. These results indicate that the main bottleneck in AI research taste lies less in finding the directions a community works on than in turning them into the concrete approaches it goes on to pursue. We release IdeaTaste as an open, continuously updated leaderboard (https://cxcscmu.github.io/IdeaTaste) that benchmarks contemporary AI proposals against future publication cycles.