Strong agents need minimal harnesses for autonomous machine learning engineering
How Much of a Harness Does a Strong Agent Need for Autonomous ML Engineering?
Artificial Intelligence
Summary
Strong AI models called large language models (LLMs) can write code and manage machine learning tasks. Usually, people add extra layers or helpers around these models to improve their work, but this paper shows that these additions don't actually help much if the AI model itself is strong and has direct access to the computing environment. The authors found that a simple setup using a single coding agent performs about as well as complex systems with multiple helpers. So, the most important factor is the strength of the base AI model, and complex extra tools don’t add much value right now.
What this means in practice
- •For machine learning engineers: Design simpler autonomous ML tools by focusing on strong base models with direct environment access rather than building complex orchestration layers.
- •For automated software developers: Create efficient code generation agents that work well with minimal infrastructure, reducing development overhead for AI-driven coding tasks.
Authors
Kirill Brilliantov, Alejandro Hernández-Cano, Emmanuel Abbé
Abstract
Recent autonomous machine learning engineering (MLE) agents have made significant progress on public leaderboards. Often motivated by progress stagnation over long-horizon cycles and limited Large Language Model (LLM) primitives, modern MLE agents are deployed on top of increasingly elaborate machinery: multi-agent orchestrators, dedicated retrieval subagents, and more. While such harnesses expand, the use of more primitive but improved coding agents - where LLMs have direct access to the execution environment through read, write, and bash primitives - has received little attention in the field. In this paper we find that, under an equal time budget and the same frontier LLM backbone, open-source state-of-the-art harnesses provide no advantages over a single session of a minimal-harness coding agent baseline, pointing to the backbone as the primary driver for performance. Via a series of large-scale systematic ablation studies, we argue that the machinery layers become redundant in the coding agent setting. We conclude that the effort spent elaborating hand-crafted harnesses around strong models yields poor returns for current MLE benchmarks.