Compact evaluation method cuts large language model test cost drastically
AMLworld-Compact: Importance-Weighted Downsampling for Cost-Effective LLM Evaluation and Error Diagnosis
Computation and Language
Summary
Testing large language models on huge datasets can be very expensive because each test requires a complex prompt. The authors created AMLworld-Compact, a smaller but smartly chosen test set that includes all suspicious cases and a balanced sample of normal cases. This lets them estimate performance metrics like precision and recall accurately while using much less data. They also show that some simple checks aren’t enough to confirm model predictions reliably. The authors provide their datasets and tools openly for others to use.
What this means in practice
- •For machine learning engineers: Reduce the cost of evaluating large language models by using stratified downsampling with importance weights to maintain unbiased accuracy estimates.
- •For fraud detection teams: Improve error diagnosis of illicit transaction detection models by focusing evaluations on all illicit cases combined with stratified normal samples.
- •For compliance monitoring providers: Offer evaluation services using cost-effective test sets that retain illicit cases fully, enabling more efficient audits of regulatory models.$Commercial implications: This paper enables selling optimized model audit tools that validate compliance detection with fewer resources.
Authors
Natapong Nitarach, Phume Ngampornsukswadi, Pittawat Taveekitworachai, Surapon Nonesung, Warit Sirichotedumrong, Kunat Pipatanakul
Abstract
Evaluating large language models (LLMs) on AMLworld's smallest test splits is costly because each of over a million targets requires a separate graph prompt. Sampling reduces this cost, but small balanced subsets change unweighted precision and F1. We introduce AMLworld-Compact, an evaluation set that retains every illicit test transaction and samples benign transactions from supervised-score strata. Its HT-Coreset protocol reduces evaluation targets by 271x and 611x on HI-Small and LI-Small. Horvitz-Thompson weights give unbiased full-test confusion-count estimates for fixed predictions; recall is exact, while precision and F1 are ratio estimates. In a held-out study using Graph Feature Preprocessor (GFP) features, one boosted-tree family defines strata for evaluating the other over 500 draws per configuration. At the released sizes, F1 RMSE is 0.00-2.58 percentage points versus 4.44-7.41 for uniform sampling with HT weights; conservative intervals remain broad. Under the evaluated graph prompts, seven open-weight LLMs have weighted precision near the predict-all-illicit baseline. In an outcome-stratified audit of 1,000 HI-Small few-shot outputs from one run per model, lexical checks pass for 84.5% of correct and 56.2% of incorrect binary verdicts, so these markers do not establish correctness. We release the evaluation sets, construction code, and audit tools at https://huggingface.co/datasets/typhoon-ai/AMLworldCompactEval and https://github.com/nat-nischw/AMLworld-Compact.