From Confusion to Clarity: Confusion-Aware Retrieval and Knowledge Injection for Text Classification
2026-09-01 • Computation and Language
Computation and LanguageArtificial Intelligence
AI summaryⓘ
The authors explain that large language models find it hard to correctly sort texts when many labels are very similar because the differences are specific to the topic and not learned during training. Common methods try to narrow down label choices by similarity but still struggle when two close labels appear together. They propose a new method that spots which labels the model confuses, adds those tricky labels to the choices, and creates simple rules to help tell them apart. This approach works without extra training and can even help smaller, less powerful models perform better. Their tests show noticeable improvements in classification accuracy on several datasets.
large language modelstaxonomy classificationlabel similarityembedding similaritypromptingconfusable labelsrule generationcross-model transferfine-tuningmacro F1 score
Authors
Manish Gupta, Chaitanya Giri, Jayasimha Talur
Abstract
Large language models (LLMs) struggle to classify text into taxonomies with many semantically similar labels, as the distinctions are domain-specific and not captured by pre-training. To handle large label spaces, a common approach retrieves top-$K$ candidate labels by embedding similarity and prompt the LLM to choose among them. However, top-$K$ retrieval reduces the number of candidates but does not help the model tell similar ones apart. When two similar labels both appear as candidates, the model lacks the signal to choose correctly between them. We propose a framework that (1) identifies which label pairs the model struggles to distinguish, (2) expands the candidate set to include confusable labels, and (3) generates targeted rules to differentiate between similar candidates. The framework requires no fine-tuning, and the generated rules transfer to smaller, cheaper models. On three benchmarks (WOS, Flipkart, LEDGAR), our approach improves Macro F1 by up to 10.0pp over retrieval baselines, with smaller models (2B--20B) gaining up to 11.5pp via cross-model transfer.