Semantic knowledge changes affect model prediction accuracy differently
Beyond Correctness: Evaluating Semantic Knowledge in Cross-Table Transfer
Machine Learning
Summary
When computer models learn from tables with different designs, adding extra meaning or context (semantic knowledge) is supposed to help prediction. But this paper shows that simply changing or hiding that meaning can make the model perform worse, even if the model didn't really benefit from that knowledge initially. The authors suggest measuring two things separately: how sensitive the model is to changes in meaning, and how much the meaning actually improves prediction compared to not having it. They also found many past studies mixed these up which can mislead conclusions about the true value of semantic knowledge.
What this means in practice
- •For machine learning engineers: Design better experiments to accurately measure when semantic knowledge truly improves model predictions in heterogeneous tabular data.
- •For data integration teams: Improve strategies for handling data with different formats by distinguishing between sensitivity to and benefit of added semantic information.
Authors
Seokyong Sheem, Hochang Lee, Suyeong Lee, Daekyum Kim
Abstract
Semantic knowledge is increasingly used to bridge heterogeneous schemas in tabular learning, but how much does that knowledge actually improve prediction? Studies in tabular learning commonly answer this question through semantic ablations that modify or suppress the supplied semantic knowledge. We show that these ablations can lead to misleading conclusions about predictive benefit: poor performance under altered semantics may be taken as evidence that the intended knowledge is beneficial. Across real and controlled experiments, altering semantic content can produce large performance differences even when the model gains little predictive benefit from having that semantic knowledge in the first place. To separate these effects, we distinguish two quantities: content sensitivity and predictive utility. Content sensitivity measures the change in performance when semantic content is altered, whereas predictive utility measures the benefit of the intended semantic knowledge relative to a suitable reference without that knowledge. This distinction motivates an evaluation framework in which the control is chosen according to the question being asked: altered controls assess sensitivity to semantic content, whereas claims that semantic knowledge improves prediction require a suitable reference. Even then, predictive utility is not fixed; it varies across suitable references and decreases when the reference can more easily recover the tested knowledge from other inputs or labeled examples. In a bounded audit of 25 semantic-ablation comparisons across nine studies, only one of 18 explicit predictive-utility claims is paired with a control that clearly isolates the tested semantic contribution. Together, these findings motivate a simple evaluation principle: semantic-ablation controls should be chosen and interpreted according to the question they are intended to answer.