Support set target leakage harms relational in-context learning models

Support-Set Target Leakage in Relational Foundation Models during In-Context Learning: Impact, Detection, and Mitigation

Artificial Intelligence

Summary

When computer models learn from example data that includes hidden clues about the answers they need to predict, they can get confused or make mistakes. This paper studies a special problem called support-set target leakage, where some information hidden in the example data (support set) is not available in the new data (queries). The authors created different types of hidden clues to see how much models rely on them and used a technique to detect and remove these clues. They found that hidden clues in the same table as the target cause the most problems, but clues that are further away in linked tables have less impact. Their detection method helps recover some model performance by removing suspicious information.

What this means in practice

  • For database developers: Identify and remove target-leaking columns in support data to improve relational model predictions during in-context learning.
  • For machine learning engineers: Use Integrated Gradients to detect and mitigate target leakage effects in relational in-context learning models on complex databases.

Authors

Roshan Reddy Upendra, Alexandre Dorais, Joe Meyer, Andrew Pouret, Anastasios Lambrianos Stappas, Dinesh Katupputhur Ramprasath, Tom Palczewski, Minghua Li

Abstract

Relational in-context learning (ICL) conditions predictions on the labeled support examples and their linked tables, creating a failure mode when the support set contains target-derived features that are unavailable for the query. We formulate this problem as support-set target leakage, distinct from leakage during dataset construction, temporal splitting, or representation learning. Here, the target-derived (leaker) columns are present only in the labeled support set during relational in-context inference, while queries remain clean. We construct 14 synthetic leaker types, corresponding to 20 columns, spanning proxies with different noise levels, coverage, modalities, semantic transparency, and relational distances. We evaluate a frozen relational encoder with an ICL head on held-out RelBench databases and use Integrated Gradients (IG) to rank and remove suspicious columns. Our results show that the effect of support-set leakage varies across tasks and relational distances. Target-table leakers cause the clearest degradation, while one- and two-hop leakers are not consistently used by the model. IG ranks target-table leakers highly across datasets and partially recovers performance in settings where leakage has the largest effect.
View PDFOpen arXiv