NoveltyScope compares papers with prior work to score scientific novelty
NoveltyScope: A Multi-Agent Research Assistant for Evidence-Grounded Novelty Assessment of Scientific Papers
Information RetrievalArtificial Intelligence
Summary
Figuring out how new and original a scientific paper is can be hard. The authors created NoveltyScope, a tool that finds earlier related work and checks how a paper’s main ideas compare. It uses multiple AI roles to debate evidence and then gives a novelty score from one to five. Testing showed the system can assess novelty more accurately than simpler methods. This shows it’s possible for AI to help judge how novel scientific ideas really are.
What this means in practice
- •For academic paper reviewers: Assist in evaluating how original submitted papers are by automatically grounding assessments in prior work.
- •For scientific publishers: Provide automated novelty scoring to support editorial decisions about which papers contribute new ideas.
Authors
Yueran Tian, Huizhi Liang
Abstract
NoveltyScope is a prototype that compares a scientific paper with retrieved prior work and produces a structured report and a five-level novelty score. The system extracts the core task and at most two author-claimed contributions, then searches a closed local corpus using BM25 and dense retrieval with reciprocal rank fusion. Contribution comparisons request paired passages for potential refutations, and quotations are checked against cached source text. Four scoring roles (Prosecutor, Defender, Judge, and Critic) use the resulting evidence, with at most one revision of the Judge's score. Evaluation uses 50 class-balanced RINoBench samples whose idea-level labels are retained for full-paper input. Mean absolute error is 1.48 and 1.26 for single-LLM and multi-agent scoring with Gemini, respectively, and 1.22 and 1.16 with Qwen3. The vector-similarity baseline has an error of 1.68. A HiCLIP case study illustrates the contribution comparisons and paired evidence passages. Overall, the results demonstrate the feasibility of evidence-grounded novelty assessment and show that the multi-agent configurations achieve lower scoring errors than their single-LLM counterparts across both evaluated models.