Long-Horizon Autonomous Architecture Research with a Language-Model Agent: A Behavioural Case Study
2026-08-03 • Artificial Intelligence
Artificial Intelligence
AI summaryⓘ
The authors studied how one large language model could act as a solo researcher to improve a vision transformer, a type of AI model for images, by running about 100 experiments on its own. They found that early on, the model made quick improvements, then hit a plateau, and only improved again when given more tools or options. The model tended to prefer small, safe changes and rediscovered known results, but also challenged some usual design choices. The study shows that how the research process is set up can be just as important as the AI's abilities.
Large language modelNeural architecture designVision TransformerImageNet-1KExperiment trackingAutonomous researchChannel attentionGreedy hill-climbingResearch workflowHypothesis testing
Authors
Aon Safdar, Mohamed Saadeldin
Abstract
We study what happens when a single general-purpose large language model acts as the sole researcher on a long-horizon neural architecture design problem. The agent receives a scientific question, an initial hypothesis and motivation, a compute budget, and research affordances (source and experiment management, experiment tracking, literature access, and persistent memory), then autonomously proposes, implements, evaluates, and records experiments over an extended period. The study comprises three phases, separated by human-declared transitions, that progressively expand the agent's tool surface or problem scale. Across approximately 100 sequential experiments, the agent improves a non-standard Vision Transformer from a weak baseline to a stronger, efficient model on small benchmarks and a usable but sub-SOTA model on ImageNet-1K, while producing a dense behavioural trace. We report four findings.(i)Productivity exhibits a clear phase structure: rapid early gains, a multi-dozen-hypothesis saturation wall, and recovery, with recovery triggered by expanding the action surface rather than changing the underlying model.(ii)A single early hypothesis contributes more to accuracy gain, with later improvements long-tailed.(iii)The preference for greedy, incremental hypotheses is largely workflow-induced: a commit-or-discard evaluation rule is isomorphic to greedy hill-climbing; the remainder reflects risk aversion after bold failures and anchoring on familiar literature. (iv)The agent independently rediscovers established results and, in the unfamiliar regime of pure channel attention, overturns a standard design choice. We conclude that workflow design was at least as influential as agent capability in this study and propose diversified search, budgeted moonshot hypotheses, explicit forks, and regime-aware re-validation as testable directions for future autonomous research.