Kernel method tests how well conditional models match real data
Conditional Kernel Stein Discrepancy
Machine Learning
Summary
Sometimes, we want to check if a prediction rule fits how real data behaves when it depends on some other information. The paper's authors created a tool that can check this fit even when they don’t know the exact distribution of that other information. Their method uses a math technique that works on many types of data, like points on curves or categories in sets. They also show their approach works well through experiments comparing accuracy and speed. This helps people test complex models in new kinds of data spaces.
What this means in practice
- •For statistical modelers: Check whether a conditional probability model fits data well even without knowledge of covariate distributions.
- •For data scientists in finance: Evaluate conditional risk models when feature distributions are unknown, supporting more reliable financial decision making.
Authors
Federico Matteucci, Florian Kalinke
Abstract
Kernel Stein discrepancies (KSDs) provide a versatile tool for comparing distributions. One of their main applications is in quantifying the goodness-of-fit (GoF) between a data-generating distribution and a prescribed target distribution. In this work, we study the related problem of conditional GoF quantification: given only a (possibly non-normalized) conditional target model, without information on the distribution of its covariates, and samples from a joint distribution, the goal is to assess how well the conditional distribution of the samples matches the target. To tackle this setting, we present a framework that allows lifting unconditional KSDs to the conditional setting through an operator-valued kernel on the covariate space, going beyond the known Euclidean case. We establish that our suggested statistic vanishes if and only if the conditional model and the true conditional distribution agree for almost all covariates and deploy it to test conditional GoF on smooth manifolds and on discrete spaces. Our experiments on level, power, and runtime demonstrate the viability of testing on these domains using the proposed statistic.