AutoResearch framework improves recommendation training with identified challenges
AutoResearch at Production Scale: Failure Modes and a Multi-Agent Framework
Machine LearningInformation Retrieval
Summary
Improving recommendation systems can take a lot of time and computer power because many options need testing. The authors used a computer program called AutoResearch that edits training steps automatically and keeps changes that work better, running it for twelve weeks on real, big systems. They found five common problems that happen when using this method at large scale and created a three-part solution to fix them. Their improvements nearly doubled some key performance measures and even created a new way to cover more content automatically.
What this means in practice
- •For machine learning engineers: Run autonomous script edits to optimize recommendation models while managing failure modes at production scale.
- •For systems reliability teams: Use the three-principle scaffolding design to detect and fix infrastructure and iteration failures during long training campaigns.
Authors
Aparajith Chandran, Juwon Kim, Saurav Jha, Pablo Castells, Florian Hottier
Abstract
Optimizing embedding systems for production recommendation pipelines demands systematic exploration that consumes disproportionate engineering effort at scale. We apply Andrej Karpathy's AutoResearch paradigm -- a large language model that iteratively edits a training script and retains modifications that improve a held-out scalar metric -- to automate this exploration. We report on twelve weeks of running this paradigm at production scale, where iterations consume hours of multi-GPU compute, evaluation involves competing criteria, and campaigns span weeks across many training jobs. Across two independently developed representation-learning systems for a book recommendation pipeline, we ran 220+ experiments and observed five recurring failure modes absent from the original setting: infrastructure fragility, agent memory decay, search-direction stagnation, iteration-cost asymmetry, and metric fixation. We contribute a three-principle scaffolding design -- prevent, persist, redirect -- that maps each failure mode to a structural remedy and whose instantiation scales with iteration cost. The framework produced a 1.82x Recall@6 lift and a 2.1x coherence lift over hand-tuned baselines, and the agent autonomously designed a text-only fallback that expanded catalog coverage by 5.8x. The two systems span nearly three orders of magnitude in per-iteration cost yet exhibit the same failure modes, suggesting these are structural properties of production-scale autonomous research rather than artifacts of either application.