SkillHorizon benchmark builds complex business tasks from smaller skills
SkillHorizon: Constructing Long-Horizon Agent Tasks through Skill Composition
Artificial Intelligence
Summary
Completing complex workflows requires many steps that depend on each other and shared information. The paper introduces SkillHorizon, a tool that creates long business tasks by linking smaller skills with clear input and output relationships. The authors test different AI language models on these tasks and find that as tasks get bigger, models struggle to finish them fully and correctly. SkillHorizon helps study how well AI agents can handle long, complicated jobs that require coordination and keeping track of progress.
What this means in practice
- •For business process automation teams: Test AI models on increasingly complex business workflows to improve task coordination and completion accuracy.
- •For customer support platform developers: Develop conversational agents capable of handling extended, dependent tasks with multiple steps based on SkillHorizon benchmarks.$Commercial implications: Enables creation of advanced support bots that manage complex customer workflows, improving service automation and scalability.
Authors
Yuxuan Liu, Zhaochen Su, Hangyu Guo, Shijue Huang, Zhixiong Zhang, Kawai Chung, Zhongwei Xie, Tianshi Zheng, Wei Fan, Qing Zong, Haochen Shi, Baixuan Xu, Haoran Li, Yangqiu Song
Abstract
Long-horizon agents must complete workflows in which later activities depend on earlier outputs and changes to shared state. A key challenge for evaluation is to expand task requirements and their dependencies while keeping tasks executable and outcomes verifiable. We introduce SkillHorizon, a benchmark that constructs long-horizon tasks through skill composition in seven business environments. Its generators use domain-specific rules to turn reusable skill procedures into concrete business activities. These activities are connected through input-output dependencies, shared resources, and state updates, allowing workload and coordination demands to grow together. Successful reference executions validate task feasibility and measure interaction workload, while programmatic evaluators assess local completion and whole-task success. We evaluate five language models on 1,050 tasks with reference interaction workloads ranging from 64K to 1M tokens. Across these scales, mean local-goal completion falls from 90.4% to 46.8%, as completed work fails to keep pace with growing requirements. At larger workloads, agents more often submit unsuccessful final answers with substantial work unfinished and tool-call allowance remaining. Trace analysis also identifies cases where scripts execute successfully but their outputs fail downstream business checks. SkillHorizon provides a configurable testbed for studying sustained progress, dependency coordination, and complete task delivery in growing workflows. Our code is available at https://github.com/KnowComp/SkillHorizon.