On-device AI assistant improves tool selection and abstention decisions
Selection Is Retrieval, Abstention Is Not: On-Device Tool Routing over 70 Korean-English Actions
Computation and Language
Summary
This paper looks at how AI assistants decide which tool to use or whether to use any tool at all when given a request, especially when working without a connection to a server. The authors find that using simpler methods to pick the best tool works well for selection, but deciding when no tool applies—called abstention—needs a smarter neural approach. They tested these choices on 600 Korean and English requests across 70 different actions and showed that neural methods improve accuracy for abstention, while simpler methods are good enough for picking a tool.
What this means in practice
- •For mobile app developers: Build AI assistants that efficiently decide whether to invoke tools locally or abstain without relying on server connectivity.
- •For embedded system engineers: Improve on-device AI responsiveness by optimizing the tradeoff between fast tool selection and accurate abstention decisions.
Authors
Janghoon Lee
Abstract
An AI assistant that calls tools makes two decisions on every request: which tool to invoke, and whether any available tool applies. In the usual design a single language model makes both, by emitting a call or by declining to emit one. On a device that has to answer without a server, the language model is what makes that design expensive, dominating both the latency and the memory of the router. The common alternative is to remove the model completely and rank the catalog of local actions with a retriever instead. That substitution is not symmetric across the two decisions. A retriever returns its highest-scoring candidate for every input and cannot signal that the catalog holds no valid action. Our earlier study found that constraining a decoder to a tool grammar repairs malformed output without improving the choice. What the substitution costs in each decision has not been measured. We evaluate the two decisions separately over 600 Korean and English requests and a catalog of 70 local actions. The router may also ask for a missing slot, reply, or delegate. Half the in-catalog requests reuse catalog vocabulary and half paraphrase it, separating lexical overlap from the action requested. Character 3-gram BM25 selects 162 of 164 lexically matched requests and 85 of 166 paraphrases. Restricting the candidate set to seven raises the paraphrase figure to a mean of 0.825 over five trials. No classifier over its score features separates in-catalog from out-of-catalog above 0.697 area under the curve, where the frozen encoder multilingual-e5-base reaches 0.806. Using that encoder for abstention alone keeps 376 of the requests local and misroutes 9 of the 150 needing delegation. Abstention, not selection, is where a neural component is required. A neural ranker improves every quality metric and is rejected on latency and memory rather than accuracy.