Large Language Model improves document clustering accuracy and speed
RAILS: Retrieval-Augmented Incremental LLM Clustering at Scale
Machine Learning
Summary
Clustering groups similar documents, but using large language models (LLMs) for this is hard because they can't handle too many labels at once and are often slow when working on documents one by one. The authors present RAILS, a method that helps LLMs cluster documents faster and better by retrieving information and adding labels incrementally while processing multiple documents in batches. RAILS showed better results than earlier methods on six public tests and worked well in a real-life software service helping with ticket topic discovery. This system can replace older clustering methods with easier control and better quality.
What this means in practice
- •For cloud service teams: Improve topic discovery in customer support tickets by replacing traditional clustering with RAILS for better quality and scalability.
- •For text analytics engineers: Scale up document clustering with LLMs using RAILS to handle large label sets and batch processing, improving throughput and accuracy.
Authors
Armin Oliya, Aleksandra Sawczuk, Radosław Białobrzeski
Abstract
Using a Large Language Model (LLM) as the clusterer at production scale is hard: prompts cannot hold the entire label space, and per-document serial processing does not deliver the throughput real workloads require. We present RAILS, a retrieval-augmented incremental LLM clusterer that turns clustering into a simple loop over a growing label pool and scales through document batching with bounded concurrency. On six public benchmarks RAILS exceeds the strongest prior LLM-clustering method on average, lifting accuracy from 51.2% to 59.3%, NMI from 67.2% to 74.8%, and ARI from 45.4% to 54.7%. We further report production-deployment evidence from a SaaS ticket-topic-discovery pipeline, where RAILS has replaced a traditional HDBSCAN stage with higher clustering quality, transparent prompt-driven control, and stateful incremental operation.