Frontier large language models show varied creativity in research ideas
Measuring the Creativity of Frontier LLMs in Automated Research
Computation and Language
Summary
Large language models, or LLMs, are getting better at doing research tasks on their own, but it's not clear how creative they are. The paper introduces ways to measure creativity by checking if ideas are useful and different from previous ones in several ways. The authors found that while models are similar in overall creativity, they differ a lot in exploring new variable combinations, which is linked to better research results. This helps understand which creativity aspects matter most for automated research.
What this means in practice
- •For ai developers: Design LLM-powered research assistants that generate more diverse and effective research ideas by focusing on expanding variable exploration.
- •For data science teams: Evaluate AI-generated hypotheses for usefulness and novelty to prioritize promising avenues in automated data-driven studies.
Authors
Yiheng Zhao, Mengzhuo Chen, Chengming Hu, Pengyi Liao, Yiran Pang
Abstract
Frontier LLMs are increasingly capable of conducting automated research, yet their creativity in this setting has not been systematically evaluated. In this paper, we propose a set of metrics to evaluate creativity along the two dimensions of valueness and novelty. Valueness assesses whether each proposed idea is useful, while novelty is evaluated from three perspectives: whether the same idea has appeared before (Exact-Match P-Novelty), whether a previously unexplored variable or variable combination is explored (Variable-level P-Novelty), and whether the idea directly follows retrieved external knowledge or departs from it (H-Novelty). Our evaluation shows that the models achieve relatively similar scores on most creativity metrics, but differ substantially in Variable-level P-Novelty, which reflects the breadth of research-space exploration. Further correlation and idea-level performance analyses show that Variable-level P-Novelty is the creativity dimension most strongly associated with research performance.