Papers for

document classification teams

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

Active learning improves rare class detection despite initial bias

Active Learning for Finding a Needle in a Haystack

Abstract: Training a classifier to find a rare class of documents, such as hate speech directed at a particular religious group, requires labelled examples that are expensive to obtain when the class makes up only a small fraction of a corpus. Active learning reduces this cost, but it must start from an initial labelled set that contains examples of the target class. Random sampling cannot supply these at reasonable cost, so they must be selected in some other way, such as keyword search, and any such selection biases the initial set towards part of the target class. We evaluate whether active learning can correct this bias. On AG News and Measuring Hate Speech, at a prevalence of around 1%, we compare random sampling with eight active learning strategies, starting from initial sets that cover or omit known subgroups of the target class, and measure recall on the omitted subgroups as well as F1. On AG News, almost every strategy recovers the omitted subgroup. On Measuring Hate Speech, uncertainty-based strategies recover less of the omitted subgroups than random sampling, whereas discriminative active learning recovers 55% of them without a significant loss of F1. Aggregate metrics conceal these differences.

Sat 10 OctComputation and Language
The gist
Finding rare examples, like hate speech, in a large collection is hard because there are very few of them to learn from. The researchers studied special methods called active learning to see if they can better find these rare cases even if the starting examples are biased or incomplete. They found some methods recover missing groups well on news data, but performance varies on hate speech data. Their work shows that different active learning methods handle initial bias differently and that overall scores can hide these important differences.
Open → 2610.15165v1

TF-IDF and BM25 score relevance as exact probability differences

TF-IDF and BM25 Are Exact KL Divergences

Abstract: TF-IDF and BM25 are two of the most widely used methods for scoring query-document relevance, yet neither has a standard probabilistic derivation that justifies it as a statistical method within a unified framework. We address this gap by showing that both scoring methods admit an exact interpretation as Kullback-Leibler divergences between two probability models. We treat the BM25 variant that includes the plus 1 correction in the IDF term, which is the one used in practice, and also discuss the original BM25 formulation without that correction. The resulting framework provides a common theoretical basis for TF-IDF and BM25, clarifies what they measure, and allows them to be compared theoretically with other information retrieval methods rather than only experimentally.

Sat 12 SeptInformation RetrievalComputation and Language
The gist
Scoring methods like TF-IDF and BM25 help computers find documents matching a search query. Until now, there was no clear probability-based reason why these scores work so well. The authors show that both methods can be exactly seen as measuring differences between probability models, known as Kullback-Leibler divergences. This gives a unified mathematical explanation for these popular techniques and helps compare them with other search algorithms.
Open → 2609.14016v1