Multi agent llm firms face challenges adapting across decades
Frontier Autolab: Organizational Memory, Adversarial Dissent and Temporal Leakage in Multi-Agent LLM Firms Across Fifty Years of Technological Change
Computational Engineering, Finance, and ScienceArtificial Intelligence
Summary
The paper studies how a group of AI roles working together like a company handles changes across 50 years of different technologies. The authors created a test where a simulated firm had to make decisions in nine different time periods from 1990 to 2040, using a shared memory and critics. They found that while the firm could often see upcoming changes, it struggled to commit to big shifts, usually sticking to what its current assets could support. The study also shows challenges in trusting such long-term simulated evaluations because of biases in scoring and memory effects.
What this means in practice
- •For enterprise ai developers: Design AI systems with organizational roles that better adapt over long time horizons despite changing technology.
- •For business strategy teams: Analyze decision-making gaps in recognizing future shifts versus committing resources to new directions in dynamic markets.
Authors
Bravish Ghosh
Abstract
Multi-agent LLM systems are increasingly structured like organizations, with roles, critics and shared memory, yet they are evaluated on tasks that last minutes. We ask how such an organization behaves when the ground it stands on keeps moving. Frontier Autolab is a long-horizon testbed in which one simulated firm, voiced by sixteen role personas and a dedicated Red Team, must re-found itself in nine technology eras from 1990 to 2040. Each era is temporally gated: the firm decides from a dated briefing, a historian-judge then reveals what happened and scores the decision on a five-dimension rubric, and lessons enter a persistent Playbook. Six eras are scored against history, one against the live market and two are open forecasts. Across four trajectories (36 era decisions, 180 subscores) we find a consistent foresight-commitment gap: in all 24 historically scored eras the judge rated the firm's recognition of the coming shift above its choice of where to build (mean gap 1.9 points on a 10-point scale), because boards chose the layer their existing assets could reach. Organizational design shaped long-run character. A Red Team armed with numeric kill gates produced fifty years of gated pilots and no product, and the rubric rated this firm highest; firms whose memory stored market-structure lessons pivoted every era, while a firm whose memory stored only validation procedure kept one method throughout. We also show why such results are hard to trust. Scores rise across eras in every run while the judge's own hindsight subscore falls (within-run r = -0.58), so apparent learning is confounded with recall of history, and we trace further distortions to self-judging, briefing selection and score aggregation. We release all records and an API harness, and specify fictional and post-cutoff eras that would turn the testbed into a benchmark.