Large language models may rely on shortcuts for causal reasoning tests
Evaluating Causal Reasoning in LLMs: A Case Study in the Challenges of Procedural Generation
Artificial IntelligenceMachine Learning
Summary
This paper looks into how well large language models can understand cause and effect using a popular test called CLadder. The authors found that the test data has easy-to-spot patterns that simpler models can also use to do well, meaning good test scores might not prove deep understanding. They also noticed that even removing important parts of the test questions barely changes how well the models perform. This suggests that the models might be guessing based on shortcuts rather than truly reasoning about causes.
What this means in practice
- •For machine learning engineers: Detect when causal reasoning benchmarks might be biased by superficial data patterns to improve model evaluation processes.
- •For ai product testers: Evaluate the true causal reasoning ability of language models beyond benchmark scores by testing robustness to input changes.
Authors
Justin Clarke, Isha Karn, Patrick Murray, KJ Hill, David Jensen
Abstract
Large language models (LLMs) appear to represent and reason about a large range of phenomena. In particular, recent studies have examined whether LLMs can accurately reason about causation. One of the most commonly used benchmarks for evaluating formal causal reasoning in LLMs is CLadder, a framework for procedural generation of benchmark data sets that was released in 2023. Experiments with CLadder appear to indicate that LLMs can perform formal causal reasoning in a large percentage of generated examples. In this paper, we examine what can be reliably inferred when LLMs perform well on test sets produced by CLadder. We identify several examples of superficial and highly predictive patterns that the CLadder generator embeds in data sets it generates. We find that even simple models such as random forests perform well on CLadder-generated data sets after training on a sample of generated data. We also find that the performance of many commercial LLMs on CLadder test sets is extremely brittle and that removing large sections of the text that are critical for formal causal reasoning has almost no effect on performance. Given these results, the most likely hypothesis for the strong performance of state-of-the-art LLMs on CLadder is that they are exploiting superficial patterns learned from exposure to training data from the generator. Any inference that LLMs are exhibiting more complex causal reasoning must rule out this hypothesis.