Retrieval guided fine tuning improves risk under noisy data conditions
Retrieval-Guided Fine-Tuning as Noisy Estimation: Risk bounds and Architectural Analysis
Machine Learning
Summary
Training machine learning models often uses data retrieved from other sources, but this data can be noisy or imperfect. The authors studied how such noisy data affects the training process using math models and found conditions when this approach works better than training only on target data or all available data. They also analyzed different ways to combine this retrieved data, showing that some approaches can become unstable due to how they weigh the information, not just because of noise. Their findings help understand when and why using retrieved data during training is beneficial.
What this means in practice
- •For machine learning engineers: Assess and select fine tuning methods for models incorporating retrieved noisy data to reduce prediction risk under varying noise conditions.
- •For natural language processing teams: Design better attention mechanisms that avoid instability due to uniform aggregation when handling noisy retrieved context in text models.
A theory result. No direct application yet.
Authors
Bhargav Lad, Yifan Hao
Abstract
Retrieval-Guided Fine-Tuning (RAG-FT) incorporates retrieved data directly into the training objective, but the statistical consequences of noisy retrieval during training remain theoretically undercharacterized. We study this question by modeling RAG-FT as an estimation problem in a multi-task linear regression framework, using an OLS proxy for single-layer linear self-attention to obtain finite-sample risk bounds. Under homoscedastic retrieval noise, we show that retrieval failure decays exponentially with task separation relative to noise, and derive explicit finite-sample conditions under which RAG-FT achieves lower risk than both target-only and full-corpus training. We then introduce a Distance-Proportional Noise (DPN) model, in which retrieval quality degrades with rank, and compare two estimators under the same retrieval process: the OLS proxy and the literal, uniform-weight forward pass of linear self-attention. We prove that the attention estimator's bias diverges as $Θ(n^{2q})$ even under exact retrieval, while OLS risk remains $Θ(d/n)$ for every noise exponent $q>0$. These results locate the instability not in noisy retrieval itself, but in the fixed, unweighted aggregation of the literal LSA forward pass, which reweighting by reliability empirically removes. We validate the predicted rate separation through direct simulation of the DPN model.