Pretraining and training budgets strongly influence ARC model performance
Pretraining and Test-Time Training Budgets Matter for ARC: A Controlled Study of VARC and TRM
Machine LearningArtificial Intelligence
Summary
Figuring out how well AI models can solve new puzzles with only a few examples is hard when the ways they are trained and tested differ a lot. The authors studied two AI models on the same puzzle set while carefully controlling how much training data and test-time learning they got. They found that giving models more time to learn during testing usually helps, but one model liked fewer varied examples while the other thrived on more. Also, one model did well even without pretraining, while the other relied on it more. This means how AI models are prepared and tested really matters for fair comparisons.
What this means in practice
- •For machine learning engineers: Optimize ARC model training by balancing pretraining and test-time learning resources to improve task-solving accuracy.
- •For ai benchmarking teams: Develop standardized evaluation setups that fairly compare ARC model architectures by controlling training and testing budgets.
Tested on one dataset.
Authors
Adrien Deliege, Marc Van Droogenbroeck
Abstract
The Abstraction and Reasoning Corpus (ARC) is increasingly used to evaluate whether neural networks can infer task-specific transformations from few demonstrations. Recent ARC models differ in architecture, but also in broader training and evaluation protocols, including pretraining data, data augmentation, test-time setup, and optimization budgets, making conclusions about architecture benefits debatable. In this work, we study how the performance of the vision-inspired VARC and recursion-based TRM models is affected by pretraining and test-time training (TTT) budgets under a common inductive protocol on ARC-AGI-1. We cross five pretraining budgets with four TTT budgets for each test task. Each budget specifies a task-variant count and total sample presentations, defining the controlled training budgets. We observe that increasing TTT sample presentations improves the standard ARC pass@2 metric in all settings for both models. Increasing the TTT task-variant count consistently helps only TRM, whereas VARC often performs better with fewer variants. VARC also remains competitive without pretraining, while TRM benefits much more from it. These results show that pretraining and TTT budgets affect ARC evaluation and motivate reporting at comparable budgets.