On-device caches speed up natural language commands conversion
On Device Agentic Operation Caches -- Classifier-Centric NL-to-Action Generation
Artificial IntelligenceSoftware Engineering
Summary
Using large AI models on the cloud to turn natural language into computer actions can be slow, costly, and raise privacy issues. The authors propose keeping a cache of common commands on the device itself, turning the problem into a classification task instead of generating commands from scratch each time. This method cuts costs by over half and makes responses about five times faster for repeated requests. They tested this on converting natural language to Excel formulas as an example.
What this means in practice
- •For mobile app developers: Improve the speed and privacy of apps that convert spoken or typed commands into software actions by implementing on-device caches for common commands.
- •For enterprise software teams: Reduce cloud inference costs when supporting natural language interfaces by using classification-based on-device caches for frequent software operations.
Authors
Moghis Fereidouni, Anthony Arnold, Sumit Gulwani, Mark Marron, A. B. Siddique
Abstract
Agentic AI is increasingly being embedded in software applications to provide natural language interfaces to features and functionality. In most cases these agents are powered by enterprise (100+ billion parameter) or frontier class large language models that require substantial computational resources run and depend on cloud hosted inference to handle the task of transforming natural language inputs into actionable software operations. This reliance on cloud-hosted inference introduces substantial network latency on top of LLM inference times, creates data privacy concerns, and, given the costs of running these models, can rapidly escalate expenses associated with supporting agentic features. This paper introduces a novel means of converting the NL-to-Action problem from a generative one into a classification-centric formulation via on-device operation caches. These caches allow an agentic system to handle frequently occurring classes of actions completely on-device -- reducing latency, enhancing privacy, and lowering operational costs. We show that for a classic NL-to-Formula task, generating Excel Formula in response to user requests, this approach reduces total inference cost by 56% when compared to cloud-only model-routing based inference and, on cache hits, reduces the latency to response latency by 5x.