Micro finds severe errors efficiently using mixed quality human feedback
MICRO: Multi-Fidelity Active Search for Severe Error Discovery
Machine LearningComputation and Language
Summary
Finding serious mistakes in data or software is important but checking everything closely is expensive. The authors designed MICRO, a method that uses both quick, cheap feedback and slower, detailed checks together. MICRO learns from features of each item to guess how likely it has a severe error and picks diverse items to check next by estimating how valuable they would be to confirm errors. Tests showed MICRO finds more serious errors than other strategies when budgets and costs vary.
What this means in practice
- •For quality assurance teams: Improve detection of critical errors in software or data by efficiently allocating mixed-cost human reviews guided by MICRO’s multi-fidelity approach.
- •For machine translation developers: Enhance error discovery in translation outputs by prioritizing items for human feedback using MICRO’s severity prediction and active search.
Tested on one dataset.
Authors
Orlando Leone, Niclas Pokel, Pehuén Moure, Yingqiang Gao, Roman Boehringer
Abstract
Human feedback can vary in cost and informativeness. Strong feedback can reveal severe errors but is costly, so cheaper quality ratings can help decide which items to annotate. We propose MICRO (Multi-Fidelity Impact Clustered Rollout), an active search framework that allocates a shared budget to these feedback types to maximise confirmed severe error discoveries. MICRO jointly models ratings and annotation losses conditional on item features to steer acquisition. It clusters acquisitions by their predicted impact on severity probabilities to select diverse candidates, then uses rollout to estimate their discovery value. Experiments on WMT20 English-German show that ratings improve both loss reconstruction and severity prediction. MICRO achieves the highest mean discovery count across four budget and rating cost settings, with similar performance to adapted MF-ENS in one and significant gains over all six comparison policies, including two rollout controls, in the other three $(p<.001)$.