Benchmark reveals challenges in predicting exact program outputs and states
Evaluating Exact Output and Checkpoint-State Prediction in Real Programs
Software EngineeringArtificial Intelligence
Summary
Predicting the exact final results and intermediate memory snapshots of computer programs just from their source code and inputs is hard. The authors created a new test set of real Python and C++ programs, with different input scenarios and checkpoints during loops. They tested several AI models that try to guess these outputs without running the code. Models that use reasoning do much better, but accuracy still drops for longer or more complex program traces. This benchmark helps find errors that simpler tests miss.
What this means in practice
- •For software testing teams: Improve automatic test oracles by predicting program outputs and states without execution, enabling earlier error detection.
- •For software debugging engineers: Use predicted intermediate program states to assist debugging by revealing likely variable values inside loops and checkpoints.
Authors
Xiaohong Chen, David Bucur, Chenglong Ma, Yi Zhang, Lingming Zhang, Sriram Vishwanath, Grigore Rosu
Abstract
We present a benchmark for predicting final output and checkpoint state from source and input alone. It extends CRUXEval-style output prediction with paired shorter- and longer-trace inputs and checkpoints inside and after a loop. The benchmark contains 400 cases from 371 Python and C++ programs, evaluated under seven settings from four model families without tools or code execution. Of 11,200 planned predictions, 11,151 produced gradable responses. Reasoning-enabled settings outperform their off counterparts by 33.1 to 55.2 percentage points on completed responses. The strongest setting scores 93.0% on shorter-trace final output, 77.0% on longer-trace final output, and 65.5% and 63.5% on the two state tasks; these scores also hold when missing responses count as wrong. Across 2,397 matched Python comparisons with identical source, changing to the longer-trace input yields 528 correct-to-wrong changes and 147 reversals. Source-clustered analyses preserve this accuracy gap, while adjusted Python models give no evidence of a positive incremental association between cumulative state load and error. Changed inputs and checkpoint tasks alter several factors together, so the gaps do not isolate trace length or an internal state-tracking mechanism. The benchmark exposes errors hidden by short-output scores alone.