Agreement between two local large language models improves merchant label accuracy
Curating Merchant-Matching Training Data with Two Confidence-Gated Local LLM Judges
Artificial IntelligenceMachine Learning
Summary
Matching payment descriptions to the right merchant can be tricky because some labels say no match when there might be one. The authors studied using two local large language models that act as judges to agree on labels before keeping them. They found that requiring both models to agree increases the correctness of the labels, but tuning how confident each model needs to be helps balance catching matches and avoiding wrong no-matches. This method improves purity of labeled data, which is important for training better systems but still needs testing for real-world usefulness.
What this means in practice
- •For payment platform engineers: Improve training datasets for merchant identification to reduce false matches and no-matches in payment processing systems.
- •For data quality teams: Set confidence thresholds for dual-model agreement to better audit and curate large-scale matching data for entity resolution.
Authors
Donghao Huang, Jinling Pei, Zhaoxia Wang
Abstract
Merchant matching resolves a noisy payment descriptor to a retrieved merchant entity or returns no match. A key challenge in curating training labels is distinguishing teacher abstention from evidence that no acceptable entity exists: false no-match labels contaminate pseudo-labeled data, while conservative labeling reduces coverage. We investigate whether agreement between two local large language model judges improves pseudo-label reliability. A label is retained only when the judges agree, with separate ordered thresholds for selections and abstentions that guarantee disjoint positive and negative label sets. Retrospective replay on 2,000 expert-annotated queries shows that higher selection thresholds can improve positive-label purity, whereas higher abstention thresholds increase false no-match labels. At thresholds (0.86, 0.80), Muse Glimmer 30B and Gemma 4 31B jointly label 1,633 queries (81.7% coverage) at 96.88% purity; positive and negative purities are 99.47% and 93.38%. This exceeds either constituent model at the same thresholds by more than two percentage points, with lower coverage. A split-half check finds only 0.14 percentage points of threshold-selection optimism. A symmetric threshold of 0.86 adds 40 erroneous no-match labels, while 46 false abstentions persist even with no confidence threshold. Across five matched within-model comparisons, higher reasoning effort yields no clear F0.5 gain and increases median latency by 1.8-5.0 times. These results motivate separate thresholding and auditing for positive and negative pseudo-labels. The study establishes label purity, not student utility; fresh-data curation and student fine-tuning remain necessary to demonstrate downstream value.