LLM based routing improves evidence screening in manufacturing supplier searches
Benefit-Targeted Routing and Bounded Revision for LLM-Based Evidence Screening in Manufacturing Supplier Discovery
Artificial Intelligence
Summary
Finding the right suppliers in manufacturing means checking a lot of information carefully. This paper studies how to use language models (like AI systems) to help review this information more efficiently and accurately. The authors tested different ways to let the AI decide when to ask humans for help and when to be confident on its own. Their new method increased agreement between AI and human reviewers in some tests, but not all. They also checked how precise the combined AI and human process was in identifying matches between suppliers and manufacturing needs.
What this means in practice
- •For supply chain analysts: Help analysts decide when AI should consult humans while screening suppliers to improve review accuracy and efficiency.
- •For enterprise ai developers: Incorporate benefit-targeted routing strategies to enhance human-AI collaboration for complex document screening tasks in manufacturing.
Tested on one dataset.
Authors
Yijiashun Qi, Hanzhe Guo, Yuxuan Li
Abstract
An evidence-review system must allocate additional model judgments and control their use. We study these software decisions on 6,164 frozen manufacturing source-task pairs. Benefit-targeted routing learns recovered agreement minus newly introduced disagreement. On the original 300-case test, 60 consultations yield 202/231 agreements with two human references versus 189/218 for a logistic-risk router. An additional 300 cases were independently annotated twice using equal frozen evidence. Without retraining the historical policies, the corresponding counts are 164/190 versus 163/193; the differences do not resolve an advantage. Strict scalar-Ridge controls share preprocessing, regularization and fitting, and remain inconclusive. Quotation and requested-state replacement gates retain rejected cases and budget costs; their joint gate rejects one original and two additional consultations without changing primary outcomes. Evidence ablations, fusion and bounded updates supply further controls. A completed 200-pair identity audit yields 97.45 percent weighted merge precision after eight author adjudications. This audits a provisional map, while prospective company-disjoint performance remains untested.