Claimnov checks which parts of ai science ideas are truly new
Do AI Scientists Know What's New?
Computation and LanguageArtificial Intelligence
Summary
Figuring out when an AI-generated science idea is truly new is tricky—many checks either reject too many good ideas or miss old ones. The authors created ClaimNov, which looks at each part of a proposal to see if it's new or already known. ClaimNov can spot disguised copied work while rarely mistaking new ideas as old. This helps keep the good, building-on-old-ideas work while filtering out unoriginal stuff more fairly.
What this means in practice
- •For ai research teams: Use ClaimNov to evaluate and filter AI-generated scientific proposals by identifying which parts are novel versus known prior work.
- •For conference program committees: Improve peer review by leveraging ClaimNov’s claim-level novelty profiles to reduce false rejections of innovative submissions.
Authors
Miri Liu, Xiayu Zhao, Encheng Xie, ChengXiang Zhai
Abstract
A major challenge facing AI scientific discovery pipelines is how to determine when a generated idea is meaningfully novel. Ideally, novelty checks in these pipelines must be able to reject non-novel ideas while not rejecting too many novel ideas, and they should also be able to distinguish which parts of a proposal are non-novel, as opposed to opaquely discarding entire proposals. We design an analytical framework for evaluation of novelty checks with temporal ground truth at both the proposal- and claim-level, using four datasets of 1,060 proposals built from published papers, reviewer disputes, and generated ideas matching ones the field went on to publish. Auditing existing novelty checks with this framework, we find that checks with high rejection rates on non-novel work also tend to have high false alarms on ideas that are later realized by human researchers, and observe that checks built to evaluate proposals as a whole can point to claim-level prior art only by rejecting the whole proposal. We therefore introduce ClaimNov, a fine-grained, interpretable claim-level novelty module. ClaimNov catches 97% of disguised papers at a 1.5% false-alarm rate, and its claim-level novelty profile cuts the LLM judge's false alarms from 11.8% to 1.5%. Because its verdict rests on claim-level evidence, ClaimNov also keeps proposals that build on established contributions, rejecting only 3% of ICLR-accepted proposals with a reviewer-disputed contribution and keeping 91 of 100 novel proposals with one spliced prior contribution, against 76 for the strongest existing check. Code and data are available at https://github.com/emmatliu/claimnov.