Typed next-hop routing layer speeds tool-use agent decisions
NIOCIS: Typed-Decision Next-Hop Routing with Node-Conditioned Option Spaces for Tool-Use Agents
Cryptography and SecurityArtificial IntelligenceMachine Learning
Summary
Making decisions with language models can be slow because each step requires rereading all information. The authors introduce NIOCIS, a new routing method that stores past actions and reduces repeated thinking by quickly choosing among typed options for each step. This method matched the original decisions about one-third of the time much faster than the usual approach, saving time without losing much accuracy. The system also identifies when decisions fall outside known options and improves over simpler methods by tailoring choices to each situation.
What this means in practice
- •For automation engineers: Speed up decision steps in tool-use AI agents by replacing repeated costly language model calls with fast typed next-hop routing.
- •For security operations teams: Improve automated incident response by integrating a routing layer that predicts and narrows action choices based on past episode data.
Authors
Jin Han, Mingfeng Xin, Feng Liang, Jingtao Yu
Abstract
Tool-use agents pay per decision: at each hop a language model re-reads the state and thinks for seconds, though its actions are finite and decisions recur. We present NIOCIS, a routing layer that folds execution logs into a loop-collapsed action graph (forward transitions plus recovery back-edges) answering one typed question: which declared option at a state, plus stop, does the fleet take next? A small model reads a calibrated distribution over them in one pass, emitting no text. On 1,615 production security-operations episodes we measure what the layer replaces, returns and is bounded by. Replaces: the typed encoder matches the logged next hop on 32.2% of the 3,875 held-out decisions (34.9% within the declared options) in 10 ms of encoder time (19.5 ms for the typed-choice form), against 2.2 s per LLM hop; two feature readouts reach 32.8% and 32.3%, an ordering the paired test does not resolve, so the typed layer is chosen for cost, not a separated lead. Tuning: jpt-4b reaches 9.9% zero-shot and 31.9% fine-tuned, a gap an exact paired test does not resolve (p=0.68), at 244 ms. Returns little: the realised hop falls outside the declared options for 7.9% of decisions and outside the model's top twenty for 9.3%; the most confident 15% holds 91.9% accuracy. Bounded: a cross-trace leave-one-out test puts the released state's ceiling at 25.6--44.9% as the state key grows more specific, i.e. repeated states disagree more often than they agree; the option-space ablation attributes the gain to per-node candidate restriction (+3.5 points over a flat catalog) rather than graph traversal (zero within its interval); a gated self-evolution loop lifts every arm above its anchor yet ends below full supervision; a confidence-gated cascade loses agreement when the escalation arm is the zero-shot stand-in; and agreement past a third of the fleet needs a state carrying the plan and the observation.