Domain-Agnostic Neural Topic Modeling with Contextual Token-Level Semantic Graph Representation
2026-08-17 • Computation and Language
Computation and LanguageMachine Learning
AI summaryⓘ
The authors found that current topic models using pre-trained language models (PLMs) struggle to clearly understand specialized vocabulary because these terms get grouped too closely together. They propose a new method called DARTopic, which builds a special graph at the token level to better capture context within documents, improving how topics are identified. This method works across different fields like general text, biomedical, and legal data without needing to retrain the large language model itself, making it more efficient and accurate at grouping related topics.
neural topic modelspre-trained language modelsembedding spacetoken-level graphsgraph neural networkstopic coherencedomain adaptationfine-tuningsemantic structuredocument clustering
Authors
Seung-Won Seo, Won Ik Cho, Yongmin Yoo
Abstract
Recent advances in neural topic models with pre-trained language models (PLMs) have achieved strong performance by leveraging general-domain pre-training, yet their topic interpretability often degrades on specialized corpora. This limitation primarily stems from the geometry of the embedding space, where domain-specific terms unseen during pre-training collapse into an indistinguishable region, and neither domain-specific re-training, word-level graph enrichment, nor parameter-efficient fine-tuning can restructure this space without inheriting the capacity ceiling of the underlying encoder. Our key insight is that a learnable graph layer operating on token-level PLM embeddings can acquire corpus-specific semantic structure that the frozen encoder lacks, because token-level graphs preserve document-local context that word-level representations discard and joint optimization with the topic objective reshapes embedding geometry directly from target-domain evidence. We instantiate this insight as DARTopic, a domain-agnostic framework that constructs token-level semantic graphs from frozen PLM embeddings and jointly trains a GNN encoder with topic inference. Across three benchmarks spanning general, biomedical, and legal domains, DARTopic consistently outperforms strong baselines in topic coherence and document clus- tering without any encoder fine-tuning, while demonstrating robustness to PLM choice and favorable runtime efficiency over fine-tuning based alternatives.