Two-track framework improves language agent adaptation and learning
TTSE: A Two-Track Online Self-Evolution Framework
Machine LearningArtificial Intelligence
Summary
This paper tackles the challenge of making AI language agents better at learning and adapting continuously in changing environments. Instead of treating what they know about the world as fixed, the authors propose a system called TTSE that lets agents update two kinds of knowledge separately: facts about the environment, and how to do tasks based on those facts. This helps agents make smarter decisions and adapt more quickly to new tasks. Tests in different simulated environments and benchmarks show TTSE works better than some previous methods.
What this means in practice
- •For ai system developers: Improve online learning and task adaptation in interactive AI agents by incorporating dual-track knowledge updating.
- •For robotics software teams: Enhance autonomous robot control by allowing separate evolution of environmental facts and task procedures for better long-term autonomy.
Authors
Ruimin Pei, Yongkang Wu, Shangyi Zheng, Yaqing Zhang, Deyang Li, Jianjun Tao, Xinyu Zhang, Xiang Zhang
Abstract
As Large Language Model (LLM) agents are applied in continuously interactive environments, driving the evolution of their own capabilities becomes a core problem for achieving long-term autonomy. Currently, environmental knowledge is typically treated as an external fixed input rather than as part of the agent's ongoing evolution. Reinforcement learning methods usually optimize policies through environmental interaction but tend to adapt only to fixed task distributions or single environments. This paper proposes TTSE (Two-Track Self-Evolution), a dual-track online self-evolution framework that separates evolving knowledge into FACT (environmental facts, whose reliability is continuously verified through interaction evidence) and TIP (task-conditioned implementation procedures). From a decision-theoretic perspective, we decompose the agent's excess risk into environment-representation regret and conditional-execution regret, characterize the conditions under which environment-conditioned policies strictly outperform condition-agnostic policies, and bound the downstream risk in terms of FACT identification error and cross-condition mismatch cost. In practice, TTSE's ablation experiments on GDPevo validate the advantage of dual-track evolution. On the classic agent task benchmarks ALFWorld and ScienceWorld, TTSE further demonstrates superior task adaptation. Moreover, TTSE is broadly compatible with existing skill self-evolution methods; combined with the Bayesian-Agent algorithm, a single-track ablation validates the dual-track advantage, substantially improving the aggregate score across the five major domains of SOPBench over three independent repetitions. Finally, on the real end-to-end task benchmark PinchBench, TTSE is integrated into a general agent framework via retrieval-based injection and stably outperforms the baseline across three independent runs.