Large language models struggle to manage vending machines profitably
AutoVend: Benchmarking LLM Agents on Long-Horizon Autonomous Vending Operations
Artificial IntelligenceComputation and Language
Summary
Managing vending machines involves many decisions like restocking and handling costs. The paper introduces AutoVend, a way to test how well language AI can run multiple vending machines over time through simulated operations. The authors found that most AI agents did worse than a simple rule-based system, showing that it’s hard for these models to make profitable plans that handle real-world challenges like inventory expiry and costs. They also noticed that improving small measurements doesn’t always lead to better money outcomes.
What this means in practice
- •For retail operations teams: Evaluate automated agent software on managing inventory and sales across multiple physical retail sites before deployment.
- •For supply chain coordinators: Test decision-making tools in handling procurement lead times and replenishment costs within connected physical stores.
Authors
Qiming Shi, Yikun Li, Beidi Luan, Rui Sun, Yan Ma, Jing Li, Daxin Jiang, Zuo Bai
Abstract
Large language model (LLM) agents are extending agent-mediated commerce on the Web from shopping for consumers to operating businesses for sellers. In online-to-offline (O2O) commerce, decisions are issued through Web-based platforms but realized through costly physical execution. Evaluating such agents requires going beyond individual transactions to ask whether decisions made online turn into sustained economic returns. We introduce AutoVend, a benchmark in which one LLM agent manages 20 vending machines across five site types through a Web-based operating interface for 180 simulated days. Its data-grounded simulator combines a shared warehouse, procurement lead times, paid replenishment, inventory expiration, and recurring operating costs. We evaluate five LLM agents against a rule-based inventory-control baseline and diagnose their site coordination, execution, and performance over time. Four of five evaluated LLM agents fall short of this baseline, and returns per dollar of inference cost rank models differently from returns alone. Operational analyses show that gains in individual metrics do not consistently yield stronger economic growth, as inventory and execution costs erode these gains. Trajectory analysis further reveals difficulties in grounding feasible actions, managing inventory commitments, and converting feedback into effective policy revisions.