Can Generalist Agents Automate Data Curation?
2026-06-02 • Artificial Intelligence
Artificial IntelligenceComputation and LanguageComputer Vision and Pattern RecognitionEmerging TechnologiesMachine Learning
AI summaryⓘ
The authors studied if automated coding agents can handle the repetitive task of curating training data for AI models. They created a fixed benchmark called Curation-Bench where agents can inspect data, apply policies, test them, and improve over time. They found that although agents can quickly match strong existing methods, they tend to only tweak small changes instead of trying new strategies unless guided by structured steps. With this guidance, agents developed better data-selection policies that use far less data than previous methods. This shows that while agents can run the data-curation process, they need careful direction to innovate effectively.
training data curationdata policiesAI developmentbenchmarkingvision-language modelsinstruction tuningautomated agentspolicy adaptationdata-selectionmethod scaffolding
Authors
Feiyang Kang, Hanze Li, Adam Nguyen, Mahavir Dabas, Jiaqi W. Ma, Frederic Sala, Dawn Song, Ruoxi Jia
Abstract
Curating training data is among the most consequential yet labor-intensive parts of modern AI development: practitioners iteratively propose, implement, evaluate, and revise data policies against noisy benchmark feedback. We ask whether generalist coding agents can automate this data-curation loop. We introduce *Curation-Bench*, an agent-centric benchmark that fixes the model, training recipe, and evaluation suite while giving agents command-line access to inspect data, implement policies, submit them to a fixed training/evaluation pipeline, and revise. In a vision-language instruction-tuning instantiation, out-of-the-box agents reach strong published data-selection baselines within ten iterations. However, trajectory analysis reveals a persistent *execution-research gap*: agents mainly tune local policy variants rather than explore new policy families, even when given strategy guides and paper references. Scaffolds requiring each iteration to cite, instantiate, and adapt a prior method shift agents toward method-guided exploration. The scaffolded agent autonomously composes -- without human design input -- a data-selection policy that outperforms strong published baselines at one-tenth their data budget. Overall, current agents can run the curation loop, but reliable data research requires scaffolded method adaptation, not open-ended prompting alone. Code and benchmark are open-sourced.