Summary
Predicting how influential a scientific paper will become is hard because it depends on many types of information available when it's published. The authors created LLM4Impact, a method that combines different kinds of data—like the paper’s meaning, its connections to other research, and timing—into one system that weighs each piece of evidence according to the situation. They also adjust for differences across scientific fields and publication times to better estimate future citations. Their approach works better than previous methods on a very large dataset, showing that different papers benefit from different information sources and that context matters a lot.
What this means in practice
- •For academic publishers: Improve editorial decisions by assessing a paper’s future influence using adaptive weighting of diverse evidence at publication time.
- •For research funding agencies: Prioritize grant distributions by estimating long-term scientific impact more accurately through context-sensitive citation forecasts.
Authors
Yong Cao, Markus Flicke, Haoyu He, Katrin Renz, Andreas Geiger
Abstract
Predicting the future impact of a newly published paper is challenging because it must be inferred from heterogeneous evidence available at publication time. Existing approaches often rely on a single source of information or combine multiple sources without accounting for their different predictive roles. In this paper, we present LLM4Impact, an evidence-aware method for scientific impact prediction that learns to represent, integrate, and calibrate heterogeneous information. LLM4Impact combines semantic, graph, LLM, and temporal representations, and injects graph information into a frozen LLM through continuous prefix tokens. A context aware gating mechanism adaptively weights different evidence, while a separate calibration module accounts for domain and temporal variation in citation scales. We further construct a large-scale benchmark dataset with 2 million papers, leakage-safe point-in-time heterogeneous ego graphs, temporal splits, and both year-level and month-level citation targets. Experiments show that LLM4Impact consistently outperforms strong semantic, graph, and LLM based baselines, with a 10.13% reduction in year RMSE on the in distribution test set and a 6.87% reduction under out-of-domain distribution. Our results reveal that the value of such evidence is context dependent: different papers benefit from different sources, while domain and publication time affect how evidence translates into citations. This finding motivates adaptive evidence selection and context-conditioned calibration rather than simply richer representations. We will release our code, benchmark, and an interactive web demonstration upon publication.