Trackrl manages skills to improve learning in long tasks
Trackable Skill Library Lifecycle Management for Agentic Reinforcement Learning
Machine LearningComputation and Language
Summary
Solving complex tasks often requires breaking them into smaller skills, but not all skills stay useful over time as a model learns. The authors created TrackRL, which tracks how valuable each skill is by carefully testing them one by one during learning. This tracking helps decide when to keep, retire, or remove skills from the system, making learning more efficient. Their experiments show that this approach leads to better performance and clearer understanding of how skills evolve.
What this means in practice
- •For reinforcement learning engineers: Manage evolving skill libraries in RL agents to improve performance on long-horizon tasks via systematic skill tracking and lifecycle control.
- •For software architects: Design modular AI systems that adaptively maintain and prune skills based on evidence-driven decisions about skill utility and mastery.
Authors
Xi Zhu, Yuanlin Duan, Changyi Yang, Kai Mei, Riyang Bao, Shichao Zhu, Dimitris N. Metaxas
Abstract
Skill-augmented agentic RL has emerged as an effective approach to improving LLM agents on long-horizon tasks. Despite this progress, individual skills may follow distinct trajectories as the policy evolves, with some becoming redundant after the model internalizes their capabilities, others remaining valuable as external scaffolding, and still others turning harmful. However, existing skill library systems fail to identify each skill's true contribution due to free-riding among co-retrieved skills, difficulty confounding across tasks, and the absence of skill-specific counterfactual evidence, which prevents reliable skill lifecycle management. We propose TrackRL, an agentic RL framework where skill tracking drives skill library full lifecycle management. Specifically, within each GRPO group, we turn grouped sampling into online randomized controlled experiments. Independent random injection and within-group contrasts on the same task isolate each skill's true contribution, and reliability-weighted aggregation accumulates evidence across groups. As evidence accumulates, we continuously track skill utility, which measures the skill's causal contribution as external scaffolding, and model mastery, which reflects how far the corresponding capability has been internalized. Along this line, we formulate skill lifecycle management as an evidence-gated state-transition process that governs each skill from generation and admission through protected incubation to retention, retirement, or removal, while a soft capacity bound controls library growth. Experiments show that TrackRL outperforms recent skill-augmented RL baselines in both library-assisted and skill-free evaluation, with further analyses revealing how skill value shifts during policy learning and how evidence-driven state transitions shape skill library evolution. Our code is available at https://github.com/xizhu1022/TrackRL.