Frontier ai models struggle to pick best training data sets reliably
DataSense-Bench: The First Step Toward an AI Scientist
Machine Learning
Summary
Picking the right data to train AI models is important but hard. The authors created DataSense-Bench to test if AI systems can choose the best pieces of data to improve a small language model. These AI agents can look at and analyze data but can't train the model themselves. The results showed that the agents only slightly did better than random at picking useful data, and they often disagreed on which data was best. This means current AI systems don’t yet have a reliable “sense” for choosing training data.
What this means in practice
- •For machine learning engineers: Assess AI assistance in selecting and ranking training data subsets to improve fine-tuning strategies for language models.
- •For software developers building ai tools: Build evaluation frameworks that measure AI agents’ ability to predict the usefulness of training data before model training.
Authors
Yudi Zhang, Mingyu Cao, Lu Yin, Mykola Pechenizkiy, Shiwei Liu
Abstract
As claims about recursive self-improvement (RSI) and artificial general intelligence (AGI) proliferate, we ask a simple question: do frontier AI models have a sense of data, i.e., can they reliably select the right data for training? We introduce DataSense-Bench to study this capability through the fundamental problem of data selection and performance forecasting in machine learning. We ask AI agents to select and rank candidate training subsets that can be used to fine-tune a small LLM model. Agents are allowed to inspect the data, write and execute analysis code, and run model forward passes, but can not train the model or access the actual evaluation tasks. We then fine-tune the base model on each selected subset and evaluate its post-training performance under a standardized protocol. We instantiate the benchmark in terminal problem solving and tool use, selecting trajectories from OpenThoughts-Agent and EnvScaler and evaluating on TBLite and BFCL, respectively. We then evaluate the agents along two complementary dimensions: the post-training performance of the top-ranked subset, reflecting the ability to identify high-value training data, and ranking accuracy, reflecting the ability to predict the relative performance of the selected subsets. In our experiments, selection gains over random selection are limited; agents do not reliably rank their selected groups, and ranking ability does not hold consistently across tasks: Astra identifies the best group in all three tool-use runs but in only one of three terminal runs. Analysis of execution traces on both tasks shows that agents often use similar data signals while interpreting their training value differently.