Pre-trained model size impacts causal learning performance variably
When Does Pre-Trained Capacity Help Causal Learning? From Scalar Prediction to Counterfactual Post-Training
Machine LearningArtificial Intelligence
Summary
This paper looks at how the size of a pre-trained computer model affects how well it can learn about cause and effect. The authors study two different systems that use the same base models but have different ways of learning and predicting outcomes. They find that making the model bigger helps in some cases but not always, depending on the task and how the model is used afterwards. The results suggest that it’s not just size that matters, but how well the model’s ability matches what needs to be learned.
What this means in practice
- •For machine learning engineers: Select model size and adaptation approaches that better fit causal effect estimation problems for improved prediction accuracy.
- •For automated code generation teams: Use larger backbone models with counterfactual post-training to boost code generation accuracy conditioned on treatment information.
Authors
Qi Xu, Zijie Lei, Zhifei Deng, Zhigang Hua, Luke Simon
Abstract
How does pre-trained model capacity relate to performance after adaptation for causal tasks? We study two systems built on the Qwen 2.5 family: CausalLLM, which estimates scalar potential outcomes with treatment-specific heads, and CausalGen, which generates treatment-conditioned text or code using soft prefixes and factual supervised fine-tuning (SFT), counterfactual SFT, and preference alignment. Increasing backbone size from 0.5B to 1.5B yields mixed scalar-estimation results on IHDP, but improves CodeGen pass@1 from 0.267 to 0.575 and treatment-effect score from 0.231 to 0.515 after the complete generative pipeline. Within 0.5B CausalLLM, narrative inputs reduce mean sqrt(PEHE) from 1.842 to 1.780, a 3.4% improvement over tabular inputs; the 5.8% improvement over a no-LLM baseline combines changes in representation and input format. These descriptive comparisons motivate a distinction between available and convertible capacity. They do not isolate the contribution of post-training: the two systems differ in dataset, architecture, output modality, and supervision, and intermediate-stage evaluations are unavailable. With only two small backbones and three replications, the results support a hypothesis about capacity--adaptation alignment rather than a general scaling law or a demonstrated causal mechanism.