Contextual embeddings capture scientific term meaning shifts beyond frequency

Beyond frequency measures: Can contextual embeddings capture meaning change in scientific texts?

Computation and Language

Summary

Tracking changes in how scientific terms are used helps identify important new trends but counting how often words appear may miss deeper shifts. The authors studied methods using advanced language models to look at the meaning of terms in context rather than just their frequency. They compared these methods on astrophysics and language science texts from 2010 to 2024 and got experts to judge real meaning changes. They found frequency counts slightly better at spotting trend words but embedding-based methods revealed important new concepts that frequency missed. Combining both approaches can give a fuller picture of how scientific terms evolve over time.

What this means in practice

  • For science policy analysts: Improve monitoring of emerging research trends by integrating semantic embedding methods to detect meaningful shifts in scientific language use.
  • For digital library curators: Enhance indexing and metadata for scientific literature by incorporating semantic drift signals beyond term frequency, improving search and discovery.

Authors

Jianying Liu, Kim Gerdes, Jean-Marc Deltorn

Abstract

Identifying technological trends is a core scientometric task, yet traditional frequency-based approaches struggle to capture substantial meaning shifts of domain-specific terms. We hypothesise that contextual embeddings can complement frequency dynamics to effectively track diachronic semantic change. We compare frequency and embedding-based approaches across Astrophysics and NLP corpora spanning from 2010 to 2024. Candidate terms are extracted using KeyBERT (utilizing SciBERT as its underlying language model) and filtered for significant frequency increases using Fisher's exact test. These terms are then evaluated for genuine semantic shift by domain experts to establish ground-truth labels. To quantify semantic drift, each term's contextual embedding ''clouds'' from the two discrete periods are compared using multiple metrics: cosine distance, average pairwise distance, Hotelling-type T 2 , and maximum mean discrepancy. Results indicate that frequency-based methods align slightly better with human judgments of ''trend-related terms'' than semantic metrics (Precision@50 of 0.62 vs 0.60 in Astrophysics). The two signals show a correlation of around 0.6. Several terms identified exclusively by embedding metrics (e.g., ''primordial black holes'') represent critical conceptual developments invisible to pure frequency analysis. These findings indicate that semantic metrics may capture complementary information, highlighting the value of integrating contextual embeddings into scientometric trend analysis.
View PDFOpen arXiv