Skillrep improves skill search by combining representation choices
SkillRep: Characterizing and Understanding the Representation Design Space of Agent Skill Retrieval
Information Retrieval
Summary
Finding the right skill from a big collection is hard because requests and skills are usually written differently. The authors study five ways to represent and match requests to the correct skills together in one system called SkillRep. They show that different representation methods help in different ways and combining them carefully can find skills more accurately. They also explain why some methods compete with each other while others work well together. This helps understand how to build better systems to retrieve skills using large language models.
What this means in practice
- •For large ai platform teams: Improve automatic skill retrieval in agent libraries to better match user requests with relevant capabilities at scale.
- •For software engineers building virtual assistants: Enhance retrieval modules to identify correct skills from vast skill pools for more accurate assistant responses.
Authors
Tianyi Sun, Pengfei Yao, Yanze Wang, Hongda Chen, Yunyun Han, Jun Sun, Chuanrui Hu, Yafeng Deng
Abstract
Retrieving the right skill from a large library is essential for LLM agents, yet requests and skills are written for different purposes and in different words. Existing retrievers narrow this request-skill gap with one or two representation choices each, so it is unknown whether their gains hold in combination. No prior work measures the gap itself, so it is also unclear which part of it each choice narrows. We define a representation design space by five choices -- encoder fine-tuning, query-side and document-side expansion, background projection and a lexical channel -- and evaluate them together for the first time, as modules of one retriever, SkillRep. With a 0.6B encoder and LLM expansion, SkillRep improves MRR@10 on SRA-Bench over the strongest same-size retriever by 7.02 points with the official skill pool and 8.79 with UniPool, a 111,247-skill pool we build from public registries, and is best on SkillsBench. The value of each choice depends on the others. Fine-tuning, for instance, reverses the ranking of the four training-free modules and turns them from complementary into overlapping. To understand why, we measure the gap with three quantities: background, the embedding component that does not separate skills; unsaid vocabulary, the gold skill's content a request leaves unsaid; and dilution, the information lost when a skill is encoded into a single vector. They show which part each module narrows and by what mechanism. This reveals that modules narrowing the same part compete for it, while those narrowing different parts complement each other.