Gradient bias effects from selective human edits on model training
When Edit Localization Amplifies Relative Selection Bias: Gradient Geometry, Target Mismatch, and Importance Weighting
Computation and Language
Summary
When humans edit parts of a model’s output, the examples they choose to fix may not represent all errors fairly, causing bias in training. The authors analyze how this bias changes when focusing only on the edited parts compared to keeping all data. They find that sometimes focusing on just edits can make bias worse or better depending on the math behind gradients and error patterns. The paper also shows how weighting samples properly can correct bias but that different methods aim for slightly different goals. Their work uses simulated and real human data on translation tasks to explore these effects in detail.
What this means in practice
- •For machine translation engineers: Improve training methods by understanding how partial human edits affect gradient bias and optimize feedback weighting to reduce errors.
- •For data annotation teams: Design better correction feedback collection that accounts for selective sampling bias to provide more effective model updates.
Authors
Shengwei Zhang, Haoda Dai, Yifei Li, Yuheng Song
Abstract
Human corrections identify editable spans, but the examples receiving corrections may come from a selective feedback channel. We analyze this interaction at a fixed model checkpoint by decomposing a localized gradient into edited and retained untouched components. Squared relative selection bias is a ratio of quadratics whose derivative has the sign of an explicit quadratic polynomial. Localization can increase, decrease, or nonmonotonically change this diagnostic; its direction depends on component biases and geometry. Oracle importance weighting recovers the population mean for each fixed localization objective, but these objectives have different targets. Against one common full-gradient target, we derive the finite-sample mean-squared error, an analytic optimal retention coefficient, and a fixed-clipping extension. Exact finite-population calculations and 10,000 Monte Carlo repetitions per sample size verify the identities and counterexamples. Public human-post-edit experiments use two translation directions and pretrained models, with declared synthetic selection. An English-German extension differentiates 73.89 million native parameters. In all three declared settings, hard localization has higher relative bias but lower absolute bias than full retention. Untouched-component biases are nonzero and selected component means have negative inner products, so the general criterion applies where the simple unbiased/aligned explanation fails. Output-bias diagnostics show the same endpoint ordering of relative bias across both directions, with one interior maximum. Mechanisms reuse each language's records and include a mixture; they are not independent replications. The evidence separates relative amplification from absolute gradient error and establishes estimation properties, without inferring translation-quality gains or identifying actual complaint propensities.