Hierarchy-Aware Supervised Uncertainty Estimation for Black-box LLM Taxonomic Reasoning
2026-08-24 • Machine Learning
Machine LearningArtificial Intelligence
AI summaryⓘ
The authors looked at how to measure confidence in answers given by large language models (LLMs) when sorting living things into categories. They used information from an open tool to teach simple models how to predict if each category level is correct. Compared to a usual method based on word likelihood, their approach did better at telling right from wrong, especially when needing to decide if the model should refuse to answer. Their best model considered the hierarchy of categories separately for each level, showing that understanding the structure helps improve confidence estimates.
Large Language ModelsUncertainty EstimationHierarchical TaxonomyBiodiversity MonitoringSupervised LearningMicro AUROCSelective PredictionToken LikelihoodMulti-head ModelAbstention Rule
Authors
Shuting Xie, Nathaniel Lesperance, Graham W. Taylor
Abstract
Large language models (LLMs) are increasingly used for scientific decision support, yet reliable confidence estimation remains difficult in black-box settings. We study uncertainty estimation for hierarchical taxonomic reasoning generated by a black-box LLM in a long-tailed biodiversity monitoring pipeline. Using proxy features extracted by an open-source tool LLM, we train lightweight supervised estimators with hierarchy-aware supervision to predict rank-wise correctness. Across three tool LLMs, the supervised estimators consistently outperform a token-likelihood baseline for micro discrimination and selective prediction under a single global rejection threshold, improving micro AUROC from 0.57 to 0.75--0.80. The best results are achieved by a rank-specific multi-head design (H3), suggesting that accounting for hierarchical output structure is important when a unified abstention rule is required. Our code is publicly available at https://github.com/uoguelph-mlrg/hierarchy-aware-llm-uq