CoffeeBench: Benchmarking Long-Horizon LLM Agents in Heterogeneous Multi-Agent Economies
2026-06-15 • Artificial Intelligence
Artificial Intelligence
AI summaryⓘ
The authors created CoffeeBench, a test for AI language models to see how well they manage a simulated coffee business over 90 days with multiple agents like farmers, roasters, and retailers. Each agent tries to make money by communicating, negotiating, and handling resources like cash and inventory. They found that while all tested models did better than doing nothing, some models communicated more and performed better, while one tended to avoid action despite making sensible plans. The authors share their code and data to help others study AI in complex economic interactions.
LLM agentsmulti-agent systemseconomic simulationlong-horizon tasksbenchmarkingcommunicationnegotiationinventory managementnet incomeagent behavior
Authors
Issa Sugiura, Daichi Hattori, Kazuo Araragi, Keita Ogawa, Shota Onose, Taro Makino, Teppei Usuki, Takashi Ishida
Abstract
As LLM agents become capable of increasingly long-horizon tasks, evaluating their performance in economic systems is becoming increasingly important. Unlike existing benchmarks that primarily evaluate a single agent interacting with a passive environment, economic systems are inherently multi-agent, requiring autonomous agents to communicate, negotiate, and transact while pursuing their own objectives over extended periods. We introduce CoffeeBench, a benchmark for evaluating LLM agents in a long-horizon multi-agent economy composed of heterogeneous firms. In CoffeeBench, two farmers, two roasters, and two retailers autonomously operate their businesses over a 90-day simulation, each seeking to maximize cumulative net income through communication and transactions while managing cash, inventory, and pricing. The evaluated model controls one coffee roaster, while the remaining firms are controlled by fixed reference agents. Across several recent open-weight and proprietary LLMs, all models outperform a passive baseline that takes no actions, with most achieving positive net income. Analysis of agent behavior reveals substantial differences in long-horizon economic interaction: higher-performing models communicate more actively with other firms, whereas Claude~Haiku~4.5 exhibits an idle-drift failure mode, repeatedly choosing inaction despite producing coherent assessments and plans. We release our code and agent trajectories to support future research.