science software developers

Abstract: Frontier coding agents are increasingly framed as end-to-end automated researchers, and ML for science offers a demanding test of that promise: improving a scientific model often requires domain insight beyond routine ML engineering. We introduce Agent4Sci, a long-horizon testbed for measuring how far today's agents can improve the published best models on scientific ML benchmarks, spanning 20 benchmarks across 11 domains. Rather than ranking agents on a leaderboard, Agent4Sci asks what kinds of progress agents can make on these tasks under in-the-wild conditions, and where they fall short. Empirically, today's frontier agents advance the published best on every benchmark, with Claude Opus reaching 10.4% mean relative improvement and no run flagged for academic misconduct. These gains, however, come almost entirely from a bag of ML-engineering tricks rather than research-level insight. Trajectory analysis surfaces three recurring limitations in this setting: novelty miscalibration, resource under-utilization, and rapid collapse into local engineering sweeps. Lightweight plug-in skills for research ideation, experiment management, and scheduled checklists raise Claude Opus from 10.4% to 20.1% and Codex from 4.4% to 8.1%, while yielding more diverse, domain-relevant interventions. Agent4Sci gives the community a way to track and push frontier agents from ML engineering toward domain insight on real-world ML-for-science problems. Code and data are available on GitHub (https://github.com/maszhongming/agent4sci) and Hugging Face (https://huggingface.co/datasets/MingZhong/agent4sci).

Abstract: The first sentence of this abstract--and the introduction to these proceedings--was authored by a human, but the bulk of this document was generated by an agentic AI system. In this talk, I take stock of machine learning (ML) for the LHC physics program over the twelve months from May 2025 to May 2026. The corpus is the HEPML Living Review, split at May 2025 into 1,756 earlier papers and 569 later ones. An AI pipeline surveyed the 569 abstracts, ranked them by citations, recency, theme, and collaboration involvement, and read 103 papers in full (95 from after the split, plus 8 earlier baseline papers), producing a structured note for each. The notes were then synthesized into six claims about the state of the field, checked by independent reviewer agents, and re-verified against the source papers. The headline claim is that (1) ML for high-energy physics (HEPML) stopped being a research area that builds tools and became infrastructure that the LHC physics program depends on: ATLAS and CMS now publish physics results that depend on neural networks, and the archived ALEPH data have re-entered production. The other five claims are: (2) simulation-based inference and foundation models are two revolutions starting to merge; (3) AI agents are the genuinely new front, with 47 papers in twelve months and no adopted measurement yet; (4) "do we trust it?" is the fastest-growing agenda, with one recent paper in five about uncertainty, calibration, or interpretability; (5) what is slowing down is informative, since equivariance was absorbed into a tool and model-specific phenomenology ceded ground to model-agnostic searches; and (6) theory ML crossed a capability threshold in multiple research areas. I close with what is settled, what is incoming, and what is open, and briefly discuss the concerns raised by this way of working with AI.