SkillSentry: Reliable Skill Execution for LLM Agents via Runtime Assurance
2026-08-10 • Artificial Intelligence
Artificial Intelligence
AI summaryⓘ
The authors study how large language model (LLM) agents use skills to complete tasks but sometimes fail because they don’t always follow the correct steps reliably. They introduce SkillSentry, a system that watches over the agent as it performs a skill, guiding and correcting its actions based on previous successes and failures. By learning from past experiences and updating its guidance during execution, SkillSentry helps agents do tasks more reliably and consistently. Tests show it improves success rates significantly over different agents and models.
LLM agentsskillsruntime assurancedomain-specific language (DSL)execution monitoringmulti-step reasoningtask success rateguidance refinementClaude modelsCodex models
Authors
You Lu, Xinyu Huang, Bihuan Chen, Xin Peng
Abstract
LLM agents are increasingly equipped with skills to perform complex tasks through multi-step reasoning and tool use. Although skills provide reusable procedural knowledge, agents may still execute them unreliably. Even when an agent has demonstrated the capability to complete tasks under the guidance of a skill, it may fail to do so consistently across similar tasks or repeated runs due to deviations from the skill procedure or incorrect execution of individual steps. Such instability limits the practical reliability of LLM agents. To address this problem, we propose SkillSentry, a skill-oriented runtime assurance framework built upon a new domain-specific language (DSL) for representing runtime guidance for skill execution. SkillSentry initializes the runtime guidance by combining a skill specification extracted from the corresponding skill document with execution experience mined from historical successful and failed traces. It then wraps around the agent execution loop to monitor and guide skill execution under the current guidance, while iteratively refining the guidance using newly collected traces. We evaluate SkillSentry on 15 skills across two LLM agents, each paired with two backbone models, i.e., Claude Code with Claude-Haiku-4.5 and Claude-Opus-4.6, and Codex with GPT-5.2 and GPT-5.4. Our results show that SkillSentry improves the task success rate of LLM agents by 24.1% across skills, on average, while exhibiting lower variability across repeated runs.