Causal model performance depends on observed data perspectives
What You Observe Determines How You Identify Causal Effects: Evaluating Causal Models across Observational Views
Machine Learning
Summary
It can be tricky to figure out cause-and-effect relationships using only observational data, and how well different models work depends on exactly what information they can see. The authors created a benchmark to test various causal models using the same underlying data but different views of the observations. They found that no single model consistently performed best, and some models struggled to stay reliable when the true causal effects didn’t change. They also showed that combining known causal identification methods with strong prediction models can work just as well or better than newer causal foundation models.
What this means in practice
- •For data science teams: Evaluate causal effect estimation techniques under different observational conditions to select the best approach for a given data scenario.
- •For machine learning engineers: Develop and deploy modular causal modeling pipelines that combine identification procedures with predictive models for more stable causal effect estimates.
Authors
Heejin Jung, Gyeongdeok Seo, Hoyoon Byun, Joseph Lee, Kyungwoo Song
Abstract
Causal foundation models (CFMs) pre-trained on data generated from various structural causal models (SCMs) have been proposed for estimating causal effects from observational data. However, differences in pre-training environments and evaluation protocols make it difficult to assess how their performance depends on the information available for causal identification. To enable controlled comparisons, we introduce CausalIDView, a multi-view benchmark that holds fixed SCM realization and target estimand while varying only the observational view available to the estimator. Each observational view corresponds to a distinct identification regime under the benchmark's maintained causal assumptions. Across these matched views, no CFM consistently performs best and model rankings vary substantially. Under controlled structural changes, CFMs exhibit model-specific failures to maintain stable estimates when true effects are unchanged and to track genuine effect changes. We also examine whether combining explicit identification with strong predictive estimation is effective. A modular approach that pairs a predictive tabular foundation model with regime-specific identification procedures is competitive with CFMs and outperforms several of them. These findings motivate cross-regime comparisons to assess the empirical value of CFMs.