Skill2Query: Exploiting Skill Structure to Generate Pseudo-Queries for Agent Skill Retrieval
2026-08-17 • Computation and Language
Computation and LanguageInformation Retrieval
AI summaryⓘ
The authors created Skill2Query, a method to improve finding the right tools (skills) for AI agents by better understanding the detailed info inside skill documents. They turn skill data into a knowledge graph and then generate example questions that clearly reflect the skill’s parts like capabilities and parameters. This helps the system find the best skill faster and more accurately across different tasks and large skill collections. Tests showed their method improves retrieval performance and makes agent tasks more successful.
pseudo-query generationskill retrievalskill knowledge graphquery templateparameter fillingretriever trainingRecall@1nDCG@1large language models (LLMs)offline index augmentation
Authors
Lihui Ding, Zihan Guo, Bingwei Lu, Chenyu Zhou, Yuanjian Zhou, Weinan Zhang, Jianghao Lin, Dongdong Ge
Abstract
Pseudo-query generation can alleviate the supervision bottleneck for agent skill retrieval, but existing document-level approaches typically leave the rich internal relations among capabilities, parameters, and usage examples implicit. As a result, generated queries may be topically relevant to a skill while lacking capability grounding and parameter consistency, raising the question of whether explicitly exploiting a skill document's internal structure can produce more effective retrieval signals. We therefore propose Skill2Query, a framework that first parses a skill document into a Skill Knowledge Graph and then generates pseudo-queries through a three-stage process including style mimicking, query template generation, and parameter filling. The generated queries can be used for offline index augmentation, online query expansion, and retriever training. Four benchmarks (TheoremQA, LogicBench, ToolQA, and CHAMP) are used to evaluate Skill2Query with large-scale skill candidate pools across multiple downstream applications, including skill retrieval, retriever training, and end-to-end agent execution. Using nearly 30K skills across diverse domains, we generate 700K category-diverse pseudo-queries. Skill2Query consistently improves sparse, dense, and skill-routing retrieval, with an average Recall@1 gain of 6.70 percentage points across retrieval settings. Skill2Query-generated training data also achieves the best Recall@1 and nDCG@1 among the evaluated generation baselines. Further evaluations with multiple LLM backends demonstrate that improved skill retrieval translates into higher agent task success rates. Code and resources are available at https://github.com/MatZaharia/Skill2Query.