Small prompt changes cause large errors in ai novelty judges
Old Ideas, Novel Problems: The Instability of LLM-Based Novelty Evaluation
Computation and Language
Summary
It can be hard to tell if an AI really creates new ideas or just repeats old ones. This paper shows that AI tools used to judge newness often give different answers depending on small changes in how they are asked. Sometimes, just hinting if humans already thought something was new or not makes the AI judge flip its decision. The researchers tested many AI judges and found most are unreliable at deciding if a new idea is truly original. This calls for better ways to check AI creativity.
What this means in practice
- •For product design teams: Avoid relying on current AI novelty judges when assessing new product ideas to prevent misleading evaluations caused by prompt sensitivity.
- •For ai development teams: Improve and validate AI creativity scoring tools using robust methods beyond simple prompting to enhance trust in generated ideas.
Authors
Noy Sternlicht, Simra Shahid, Peter Jansen, Daniel S. Weld, Pao Siangliulue, Tom Hope
Abstract
Automated ideation systems are often evaluated on the novelty of the ideas they produce, and that judgment is increasingly delegated to large language models. Such judges are typically built ad hoc and validated, if at all, on human-authored papers rather than on the generated ideas they are meant to score. So, how do novelty judges perform? Not well. We present a systematic controlled study of novelty evaluation design choices. We first build an evaluation set automatically, mining OpenReview for passages where reviewers explicitly affirm or dispute a paper's originality and keeping only submissions with unanimous agreement at the extremes of their research area; we pair these with ideas from a vanilla LLM generator. Across six judges, we find that small prompt design choices have large consequences; e.g., simply telling the judge that reviewers found one idea novel and the other not can change its verdict on more than half of the identical idea pairs it is shown, shifting pairwise accuracy by over 50 points and occasionally pushing it below chance. The same change helps one judge and hurts another. Retrieval and larger reasoning budgets help little, and two purpose-built novelty evaluators are outperformed by our cheapest prompted baseline. These results raise questions about reported novelty gains of automated ideation systems, and call for robust novelty evaluation methods.