Lightweight filter reduces policy checks in language agents by 30 percent
Signals and Queries: Token-Efficient Agentic Context Discovery
Computation and Language
Summary
Large language model (LLM) agents decide how to respond by choosing from many possible behavior policies, but testing every policy each time is expensive. The authors created a method called Signals and Queries (S&Q) that quickly filters out unlikely policies before doing expensive checks. S&Q trains simple classifiers using example conversations generated offline, so that during use it avoids calling large models for every policy. This approach cuts the number of policies to evaluate by at least 30% while still keeping almost all relevant ones. It helps customer-facing chatbots run more efficiently without missing important options.
What this means in practice
- •For chatbot engineers: Reduce computational costs by filtering irrelevant policies early in customer service AI agents using inexpensive classifiers trained on conversation examples.
- •For ai system integrators: Improve large language model agent responsiveness by preselecting candidate policies with efficient embedding-based classifiers before expensive model calls.
Authors
Yam Marcovitz
Abstract
User-facing LLM agents must identify which behavioral policies apply at each turn, but evaluating every policy with an LLM is costly. We introduce Signals and Queries (S&Q), a lightweight first-stage relevance filter developed for Parlant 4.0 that removes at least 30% of candidate policies before downstream evaluation while preserving high recall. For each policy, an LLM generates activating examples and hard negatives offline. An independent logistic classifier trained on their embeddings then predicts relevance using embeddings and vector dot products, with no per-turn LLM calls. This shifts expensive relevance evaluation to a smaller candidate set, while allowing each policy's classifier to be updated independently. Evaluated against 9,144 LLM-judge relevance labels covering 254 $τ^2$-bench conversation prefixes and 36 airline policies, S&Q achieves higher precision than top-K and cosine-threshold retrieval at every evaluated recall level, with consistent results across two embedding providers. These findings support inexpensive, per-policy classification as a practical first stage for reducing downstream evaluation workload in customer-facing agents.