science policy analysts

Abstract: The research idea in a published paper is rarely the project's initial idea. Between the two lies a closed-loop process in which researchers run experiments, derive insights, identify challenges, explore the literature, and revise the idea, with each step motivated by evidence and reasoning they can articulate. We call the path one idea takes through this process, with the evidence and reasoning behind each step, its authentic research trajectory. Although it underlies every scientific contribution, it is rarely recorded. Existing LLM-based research agents generate an idea once, refine it against reviews, literature, or their own experimental results, or search over candidate ideas and experiments. In all of them the text of the idea evolves, while the evidence and reasoning that connected one version to the next are discarded. We propose the Research Progress Graph (RPG), a representation of authentic and complete research trajectories. An RPG records the evolution of an idea as typed research states connected by cognitive transitions, each annotated with a temporal index and the reasoning behind it, which makes this evolution an explicit object of study. We ground the representation in authentic trajectories whose steps their authors justify in annotations and papers. Comparing the representations of thirty-five research agents with the RPG, we find that each is an incomplete RPG; none keeps the state that changed, the evidence that changed it, and the reasoning connecting them as one evolving record. Modeling ideation with the RPG changes how research agents should be designed, evaluated, and trained. Agents predict the next research state instead of a complete idea, ideas are evaluated as trajectories instead of final texts, and authentic trajectories serve as training data. These directions shift research agents from generating static ideas to modeling how ideas evolve.

Abstract: This short conceptual paper recalls to the attention of the scientometrics community some long-standing open methodological problems and proposes new directions that might be taken in renewed engagements with them. These problems are the differential dependency of the global reference score on the included units' observed scores and the dilution of impact measurement accuracy due to collaboration, which causes a global convergence in country citation impact due to international collaborative research. Advantages and disadvantages of existing and new solutions are discussed.

Abstract: Paradigm shifts in science are visible in what researchers assert and contest before they are visible in the citation graph. We ask whether a cheap, content-level signal of epistemic instability, derived from the changing distribution of stated modelling beliefs in paper abstracts, can anticipate a paradigm shift earlier than the dominant citation-based disruption index (CD5). On the displacement of recurrent networks by Transformers in NLP, a pre-registered content signal crosses its detection threshold in 2016-Q1, whereas a real-time CD5 monitor cannot even observe the 2017 breakthrough until 2022-Q2, since CD5 needs a five-year forward-citation window: a lead of about 25 quarters. The flat citation baseline is not an artifact of one index, as our CD5, a reference-normalised variant, and an authoritative precomputed index all sit near zero across the shift. The lead is also not a faster proxy: at equal latency the signal beats the content competitors tested, including a learned CD-from-text model and an embedding disruption measure, and the decomposition sees what a keyword cannot (keyword-blind AUC of about 0.9, reproduced on human labels). The lead over CD5 is an observability lead. Which signal carries it depends on the shift: belief adoption in NLP, contestation and applicability stress in vision. Measured against the breakthroughs themselves, the signal leads by five quarters in NLP and is contemporaneous in computer vision.

Abstract: Funders know their portfolios through software that classifies award text. The US National Institutes of Health (NIH) reports its spending in more than 300 categories mined this way, and work on classification and indicators treats the text as the applicant's to write. In 2025 NIH required "DEI language" removed from awards not supporting DEI activities, and so policed the words it also counts. We followed population names through 37,790 continuing awards, checking them against practice records. Names were informative: in new awards with trial baselines, a title naming Black populations predicted a 60-percentage-point higher enrolled share. Text features predicted which names survived, and practice records added little. NIH's minority health category followed the names: of continuing projects carrying it, 99.4% with unchanged summaries kept it, against 15.7% of those whose summaries no longer named a racial or ethnic population, while the projects kept their funding. Blinded reviewers judged 42 of a random 50 such losses to have met its definition in FY2024. Comparing both summaries of 70 name losses, they found aims concerning the population recast in 49. Read alone, 63 FY2025 summaries no longer met the definition. Awards naming sexual and gender minorities were recorded as terminated 37.8 points more often after adjustment for listed terms, institute and activity. We call the mechanism word targeting, its boundary population targeting, and its product uncounted science: funded research the count no longer records. The count followed the edited text, and the edited record cannot say whether the research changed with it.

Abstract: Scientific progress emerges from a longitudinal ecosystem in which researchers, institutions, funding agencies, collaboration networks, and the scientific literature co-evolve. As AI becomes increasingly involved throughout the scientific research cycle, understanding these interconnected and evolving processes becomes increasingly important. We introduce SciUtopia, a persistent, closed-loop LLM-agent simulation framework for studying academic research ecosystems. SciUtopia models interconnected scientific processes such as research-direction choice, collaboration, submission, peer review, resubmission, citation, funding, and researcher attrition, while maintaining evolving states across simulated years. Its configurable institutional mechanisms and information channels provide a controlled testbed for matched counterfactual experiments and targeted interventions. Across 61 simulation worlds, SciUtopia simulates over 40,000 researchers from 8,000 institutions, producing around 400,000 publication decisions and 1.2 million LLM-generated peer reviews. Using these longitudinal simulations, we find that rejection-driven resubmission substantially amplifies reviewer burden beyond population growth alone, cautious exploration balances citation impact with career success and long-term topic diversity, and resource inequality can emerge even without detectable cumulative advantage from narrowly winning early funding. Code is available at https://github.com/Ahren09/ScienceUtopia.

Abstract: Internationally mobile scientists are central to national research and innovation systems. We link faculty rosters from the Academic Analytics Research Center to OpenAlex publication records for 2011-2020, yielding more than 12 million faculty-publication observations for 236,394 tenure-system faculty at more than 300 major U.S. universities. Foreign-trained faculty, defined by a terminal degree awarded outside the United States, constitute about 11% of the observed faculty workforce but account for 13-14% of publications and 14-16% of top-1% cited elite output. We find that this elevated representation in elite output is closely related to their organizational embeddedness: when faculty are compared within the same institution, scientific domain, rank, and year, the difference in elite output narrow substantially while overall productivity and collaboration differences persist within those settings. In addition, foreign-trained faculty enter each publication year with larger and broader prior collaboration networks, and collaborator reach is more strongly associated with subsequent elite output. By contrast, although raw topic breadth is greater among foreign-trained faculty, after accounting for prior publication volume, we find that their topic breadth is slightly narrower and more cognitively concentrated. These findings recast international training in the lens of scientific capacity and organizational integration in that internationally accumulated scientific capabilities become embedded in institutions and relationships through which research is organized and produced.

Abstract: Identifying technological trends is a core scientometric task, yet traditional frequency-based approaches struggle to capture substantial meaning shifts of domain-specific terms. We hypothesise that contextual embeddings can complement frequency dynamics to effectively track diachronic semantic change. We compare frequency and embedding-based approaches across Astrophysics and NLP corpora spanning from 2010 to 2024. Candidate terms are extracted using KeyBERT (utilizing SciBERT as its underlying language model) and filtered for significant frequency increases using Fisher's exact test. These terms are then evaluated for genuine semantic shift by domain experts to establish ground-truth labels. To quantify semantic drift, each term's contextual embedding ''clouds'' from the two discrete periods are compared using multiple metrics: cosine distance, average pairwise distance, Hotelling-type T 2 , and maximum mean discrepancy. Results indicate that frequency-based methods align slightly better with human judgments of ''trend-related terms'' than semantic metrics (Precision@50 of 0.62 vs 0.60 in Astrophysics). The two signals show a correlation of around 0.6. Several terms identified exclusively by embedding metrics (e.g., ''primordial black holes'') represent critical conceptual developments invisible to pure frequency analysis. These findings indicate that semantic metrics may capture complementary information, highlighting the value of integrating contextual embeddings into scientometric trend analysis.