Large language models struggle to run Indian neighborhood shops effectively
KiranaBench: Can a Language Model Run a Neighbourhood Shop?
Computation and LanguageMultiagent Systems
Summary
Running a small shop involves many tasks and languages, which large language models (LLMs) find difficult to handle well. The authors created KiranaBench, a test setup that simulates a typical Indian store with four roles managed by separate LLMs speaking English, Hindi, or Hinglish. They found that the models performed much worse than a simple rule-based system, especially in Hindi and Hinglish. This shows that current LLMs are not yet good at running small businesses in India on their own.
What this means in practice
- •For small business software developers: Test and improve autonomous agents for managing Indian neighborhood shops with multiple roles and languages.
- •For multilingual ai developers: Use KiranaBench to evaluate language models’ performance in realistic, multi-agent, multilingual Indian retail settings.
Authors
Bhuvan Arora, Ankita Karmakar, Chaitanya Jain, Dilesh Churiwal, Harshit Jain, Shivansh Manish Dubey, Manish Prasad, Raja Piyush, Kshitij Tayal, Dhruv Kumar, Jagat Sesh Challa, Raunak Manekar, Yash Sinha
Abstract
Large language model (LLM) agents are increasingly evaluated on long-horizon tasks, including autonomously running a business. Existing benchmarks for this setting are almost exclusively English-language, Western in retail context, and single-agent, with the agent observing the business directly. We study how these results transfer to a multi-agent Indic setting. We introduce KiranaBench, a simulated Indian general store in which four roles (manager, cashier, helper, and bookkeeper) are each played by a separate LLM agent. Interactions with customers and suppliers occur in English, Hindi, or Hinglish, and product demand is calibrated to India's national Household Consumption Expenditure Survey. Evaluating four open-weight LLMs, we find that all fall far short of a fixed-rule shopkeeper baseline: the best model earns only one-eighth of the baseline's profit, and two models finish below an unmanaged shop. Performance is also consistently lower for Hindi and Hinglish than for English supplier communication. KiranaBench offers evidence that current LLM agents are not yet reliable for autonomously operating small businesses in India, and provides a testbed for developing agents suited to settings that are long-horizon, collaborative, and multilingual.