Tree-of-Experience: Hierarchical Experience Management for Self-Evolving Agents
2026-08-10 • Computation and Language
Computation and Language
AI summaryⓘ
The authors created a system called Tree-of-Experience (ToE) to help language model agents learn better from their past problem-solving efforts. Unlike older methods that treat experiences separately, ToE organizes experiences into a tree that mirrors how the agent thinks through problems step-by-step. This structure helps the agent remember which reasoning paths worked well and improve future performance more efficiently. Tests on two problem sets showed that ToE made the agents more accurate and faster compared to traditional approaches or not using past experience at all.
Large Language Models (LLM)Continual learningExperience managementHierarchical reasoningTree-of-ExperienceProblem-solving performanceFeedback attributionCross-task transferGame of 24FinEvolveBench
Authors
Zihao Deng, Yining Zhu, Leiming Wang, Jingfei Lu, Junbo Wang, Chuncheng Ran, Yu Yang, Dixuan Yang, Jikun Shen
Abstract
Continual self-evolution requires LLM agents to transform environmental interactions into reliable and reusable experience. Existing methods typically refine individual trajectories or abstract shared knowledge from related trajectories, but their experience representations are often disconnected from the underlying reasoning process. This limits feedback attribution, cross-task transfer, and update and retrieval efficiency, particularly in complex reasoning tasks with outcome-level feedback. To overcome this limitation, we propose \textbf{T}ree-\textbf{o}f-\textbf{E}xperience (ToE), a structured experience-management framework that aligns experience organization with the hierarchical reasoning process of LLM agents. Specifically, ToE organizes the experience into a shared tree of analytical perspectives and reasoning paths, whose reliability is calibrated through environmental outcomes to support systematic updating, transfer, and efficient retrieval. The experimental results on \textsc{Game of 24} and \textsc{FinEvolveBench} show that ToE substantially improves both problem-solving performance and efficiency. On \textsc{Game of 24}, ToE achieves a 31.4\% relative improvement in accuracy over the experience-free ToT baseline. On \textsc{FinEvolveBench}, ToE improves tsIC by an average of 41.24\% over the experience-free pipeline across 12 evaluation settings, whereas conventional experience-management methods often underperform experience-free baselines.