Fragility of rare events improves unseen species estimation accuracy
Rarity of Singletons Matters & Supplementary Materials
Machine Learning
Summary
Estimating how many unseen types or rare events exist based on small samples is hard because the rare observations are often unreliable. The authors introduce a method that measures how fragile these rare data points are and adjusts estimations accordingly to be more stable and accurate. Their approach improves traditional techniques by reducing the impact of unpredictable rare events without complicated assumptions. Tests with both made-up distributions and real language data show better performance, especially with small or skewed samples.
What this means in practice
- •For language technology developers: Improve language models by more reliably estimating probabilities of rare words from limited text data.
- •For ecological survey analysts: Produce more stable estimates of unknown species counts when sample sizes are small or biodiversity is highly uneven.
Authors
Pragatheeswaran Vipulanandan, Kamal Premaratne, Dilip Sarkar
Abstract
Estimating missing-mass and unseen-species counts from finite samples relies heavily on rare-event occupancy statistics, particularly singletons. In small-sample regimes, these statistics can be highly variable, making classical estimators sensitive to sampling fluctuations and potentially unreliable. We propose a fragility-aware (FA) framework that explicitly quantifies this sensitivity and adaptively regularizes conventional missing-mass and unseen-species estimators. Using singleton-based fragility measures derived from probability mass and occupancy counts, we formulate an entropy--Kullback--Leibler (KL) optimization that adjusts the reliance on base estimates according to their fragility. The resulting objectives are strictly concave, admit unique closed-form solutions, and require no Poissonization or asymptotic approximations. The framework provides a lightweight correction to classical estimators, including Good--Turing and Good--Toulmin, while reducing reliance on potentially unstable rare-event statistics. Across 360 experimental configurations spanning synthetic distributions and real-world language corpora, our method remains competitive in stable regimes while improving estimation accuracy and stability in fragmented, long-tailed, and small-sample settings. The findings demonstrate that accounting for the fragility of rare observations can improve the reliability of inference about unobserved species and their associated probability mass.