Rosetta creates performance models from papers without human help
Rosetta: Automating First-Principles Performance Modeling Using Multi-Agent LLMs
PerformanceHardware Architecture
Summary
Researchers often need detailed math models to understand how fast computer parts work, but building these models by hand takes weeks. The team behind Rosetta made a system that reads research papers and automatically writes these models in math and code, explaining them in simple language. It checks itself carefully to avoid mistakes and even spots hidden assumptions and missing info. This helps experts verify claims and improve their work faster. People who tested Rosetta found it useful enough to revise their experiments and want to use it again.
What this means in practice
- •For hardware architects: Generate detailed performance models from research papers automatically to speed up hardware design validation and insight.
- •For software engineers: Obtain executable models and explanations from hardware research papers to guide software optimizations without manual modeling effort.
Authors
Karthikeyan Sankaralingam
Abstract
Analytical performance models --- derivations of throughput or speedup from hardware parameters --- make claims independently verifiable and expose binding constraints, yet rarely accompany architecture papers because building one by hand takes weeks of expert effort. We present Rosetta, a multi-agent LLM pipeline that automatically generates first-principles analytical models from research paper PDFs. Given a paper as sole input, Rosetta produces a mathematical specification, an executable Python model, and a plain-English interpretation --- all autonomously, with zero human intervention. The formalization process itself is the primary value: it surfaces implicit assumptions and identifies missing parameters. Four design decisions address failure modes of naïve LLM-based generation: a scientific constitution that prohibits circular reasoning, verify-repair loops with independent critic agents, dual verification separating functional correctness from scientific validity, and a best-of-$N$ ensemble that exploits LLM stochasticity. We evaluate Rosetta across three complementary tracks: expert evaluation of 12 landmark papers (CS1), automated scoring of 97 unfiltered ISCA 2025 and HPCA 2026 papers (CS2), and author self-evaluation by six active research groups (CS3). Across CS1, specification quality scores 4--5/5 on 10 of 12 papers with zero significant hallucinations; across CS2, 56\% of fit-screened papers reach Tier~A insight quality. The strongest finding comes from CS3: Rosetta's output led to revised claims and new experiments in active submissions, and five of six author-evaluators said they would use it again.