Exact effect of adding data on Dirichlet-smoothed Markov models
Marginal Log-Likelihood Increments under Dirichlet-Smoothed Markov Estimation
Machine Learning
Summary
This paper looks at what happens when a single new data trace is added to a special kind of prediction model called a Dirichlet-smoothed Markov model. The authors found an exact way to calculate how the model's goodness of fit changes with this extra data. They also discovered mathematical limits and relationships that explain how adding more data pieces interacts. In tests on loan application logs, they showed that being able to predict individual data improvements doesn't always mean better overall batch improvements.
What this means in practice
- •For data engineers: Evaluate exact impact on model fit from adding new sequential data traces during incremental training.
- •For machine learning engineers: Use derived bounds and sign criteria to guide batch data selection for Dirichlet-smoothed Markov models.
Tested on one dataset.
Authors
Levin David Schwab
Abstract
For a Dirichlet-smoothed transition model, the effect of adding one workflow trace to the training archive is an exact change in reference-weighted log likelihood. We derive that change and show that it is a weighted reduction of Kullback--Leibler divergence between the reference conditionals and the model. From this form we obtain an upper bound on the gain available to any acquisition, which expresses a millinat difference as a share of what is attainable, an exact covariance identity for the effect of the reference weighting, and a sign criterion for the interaction between two candidates, from which the batch objective is neither submodular nor supermodular. A case study on the BPI Challenge 2012 loan-application log measures all three and finds a positive selection result in one of the four combinations of reference weighting and budget unit. There, of two regressors fitted to identical descriptors and identical labels, the one that predicts individual increments far more accurately, median $R^2$ 0.87 against 0.62, realizes the smaller share of the attainable gain, 61 against 69 per cent, so ranking accuracy for individual traces is neither necessary nor sufficient for batch quality.