Segment-level topic modeling lowers cost and improves analysis quality
Segment-Level Agentic Topic Modeling for Improved Data Exploration and Resource Efficiency
Artificial Intelligence
Summary
Discovering hidden topics in documents helps understand large text collections, but current methods using large language models can be costly and sometimes give topics that are too broad or narrow. The authors introduce SeLATM, a technique that breaks documents into smaller parts to generate and refine topics more efficiently using feedback loops. Their tests show this method saves computing resources and produces better topics than older approaches. This could help businesses analyze huge amounts of text more effectively and cheaply.
What this means in practice
- •For enterprise data teams: Analyze large volumes of business documents to find detailed themes with lower computing costs by using segment-level topic modeling and iterative refinements.
- •For content management professionals: Improve the quality and relevance of automatically generated topics in large document collections to better organize and retrieve information.
Authors
Myeongjun Erik Jang, Antonios Georgiadis, Sae Young Moon, Fran Silavong
Abstract
Topic modeling is an effective technique for discovering hidden themes within documents and is widely used in text mining and data analysis across a variety of industry sectors. Recently, large language model (LLM)-based topic models have been emerged that prompt LLMs to generate topics then assign the topics to documents, producing more natural and human-readable topics than conventional topic modeling algorithms. However, the nature of topic assignment process causes certain drawbacks, such as the incapability to produce topic distributions over a document, too broad or narrow topics, and high resource consumption, which increases with the number and length of of documents being assigned topics. These issues are particularly critical for industrial applications, which require high-quality, in-depth analysis and the processing of large volumes of documents. In this context, this paper introduces a framework called SeLATM, which addresses these concerns by employing segment-level topic generation and topic refinement through agentic feedback loops. Experimental results on various datasets demonstrate that SeLATM significantly reduces the LLM resources compared to methods based on topic assignment process, while maintaining superior performance.