Papers for

automated tuning teams

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

AI research runs unified to reveal search patterns and improve setups

ROAR: Unifying Runs across Heterogeneous AI-Driven Research Systems

Abstract: Each run of an AI-driven research system (ADRS) is an expensive search over a vast solution space, and dependable evaluation requires many runs, making run data both costly to produce and valuable to retain for large-scale analysis. Yet this data remains fragmented: teams operate in isolation, ADRS frameworks emit results in different formats, and no shared infrastructure exists to aggregate or compare runs across problems and systems. We present ROAR, a solution for systematically unifying and analyzing heterogeneous ADRS outputs. ROAR addresses two challenges: reconciling heterogeneous ADRS outputs and enabling analytics across runs with different objectives and scoring functions. We achieve this through a relational schema and parsing layer that normalize heterogeneous ADRS outputs while preserving data lineage and temporal structure, and accommodating new systems without requiring schema modifications. From building a corpus of more than 900 runs from multiple ADRS, we show how pooled data can reveal properties of problem landscapes that are difficult to observe. Consistent with prior work, runs with identical configurations may converge to different scores. We find that many runs realize most gains early, and that the effectiveness of different strategies for incorporating prior solutions into the search process varies across problems. We further show that the pooled corpus is actionable and not merely analytical by using ROAR to configure ADRS runs. Together, these results illustrate how pooled ADRS data can expose problem-dependent structure in search behavior that is difficult to detect from any single system, team, or benchmark. Such cross-cutting insights are difficult to obtain while runs remain siloed; ROAR is the first infrastructure designed to unify them.

Fri 2 OctArtificial Intelligence
The gist
Running AI research systems is costly because each run searches through many possible solutions. The authors created ROAR, a platform that collects and standardizes data from many different AI system runs so they can be compared and studied together. By pooling over 900 runs, they found that even identical setups can lead to different results and that most improvements happen early on. ROAR also helps tune new AI runs by learning from this shared data. This unified approach shows patterns that are hard to see when data is kept separate.
Open → 2610.03966v1