Text classifier results vary greatly by document version used
The record is part of the task: matched-record evaluation of text classifiers across maintenance, safety and recall reporting
Machine LearningComputation and Language
Summary
When the same event is described in different reports written at various stages, the accuracy of text classifiers changes a lot depending on which report is used for evaluation. The authors studied three real-world systems and found that differences between reports can cause bigger changes in model performance than the choice of model itself. They suggest that evaluations should consider which document version is available at the decision time and explain how the labels and records were created.
What this means in practice
- •For aviation maintenance teams: Select the appropriate report version to train and evaluate text classifiers for more accurate identification of repair events.
- •For vehicle safety analysts: Use the strongest summary document available when building classifiers for identifying vehicle defects to improve safety reporting accuracy.
Authors
Hisham Ihshaish, Peter Mayhew, Tasnim M. A. Zayet, Ana Del Amo
Abstract
Many operational cases are documented more than once, at different workflow stages and for different purposes, yet model evaluations normally select one of these records before model comparison begins. We treat that selection as part of the evaluation and compare matched records of the same cases under fixed labels and splits in three systems: GE Aerospace repair events, NASA ASRS safety reports and NHTSA vehicle recalls. Across the three GE fields, for events whose label comes from parts transactions independently of the narratives, held-out macro-F1 ranged from 0.33 to 0.91. A difference of 0.46 separated the customer report, written before shop work, from the technician report, written after diagnosis but before the transaction that generates the label. That difference is substantially larger than the representation and architecture differences tested on the same events. The public systems showed different patterns: the NHTSA defect summary remained strongest under every model family tested, whereas the ASRS analyst synopsis outperformed the reporter narrative under learned sequence models but not under lexical baselines. Secondary analyses showed that some model comparisons were also record-dependent. Evaluations should be run on the information available at the intended decision point and should report how both the record and the label were produced.