AI agents tested on real ML projects for production fixes and updates
MLOps-Bench: Benchmarking AI Agents on Production ML Engineering in Real Repositories
Software Engineering
Summary
AI systems are getting better at helping fix and update machine learning projects in the real world, but most tests only check simple tasks or toy examples. The authors created MLOps-Bench, a new set of real-world tasks based on actual machine learning projects with clear instructions and code tests. They tested six AI agents and found that while some progress was made, these systems still struggle to fully fix or update production machine learning setups. This benchmark shifts attention to real maintenance work that requires understanding how different parts of a machine learning system fit and work together.
What this means in practice
- •For ml engineering teams: Evaluate AI tools on real production machine learning maintenance tasks to understand their effectiveness and limitations.
- •For software developers: Use MLOps-Bench to test AI agents that propose and verify code changes in active machine learning repositories.
Authors
Quoc Co Tran, Hitesh Laxmichand Patel, Diwakar Mahajan, Kshitij Bakliwal, Avi Sil, Katrin Kirchhoff
Abstract
AI agents are increasingly expected to modify and maintain machine-learning systems in production, yet existing benchmarks primarily evaluate either competition-style ML experimentation or generic repository issue resolution. We introduce MLOps-Bench, a repository-grounded benchmark for production ML engineering. Unlike prior ML-agent benchmarks, MLOps-Bench evaluates executable lifecycle changes inside real ML repositories, spanning data pipelines, feature engineering, model training, model serving, monitoring, CI/CD and governance, and end-to-end integration. Each task is grounded in repository evidence and includes a natural-language requirement, oracle patch, plausible buggy baseline, fail-to-pass tests, pass-to-pass tests, and provenance metadata. The benchmark contains 212 validated tasks from 31 public ML repositories, with 636 fail-to-pass tests and 856 pass-to-pass tests. We evaluate six agents and find that the best system solves 31.2% of tasks while passing 68.7% of tests, showing that current agents often make partial progress but still fail production-relevant behavioral requirements. MLOps-Bench shifts agent evaluation beyond model building and issue fixing toward repository-native MLOps work where correctness depends on lifecycle reasoning, behavioral robustness, and preservation of existing system behavior.