No Time Like the Present: Agentic Test-Time Training for LLM Agents
2026-07-03 • Machine Learning
Machine LearningArtificial Intelligence
AI summaryⓘ
The authors study how large language model (LLM) agents can perform worse over long tasks by repeating mistakes or losing good strategies. They look at continuously updating the model during the task, but find this can cause the model to get stuck by learning from repeated errors. To fix this, they create Agentic Test-Time Training (aTTT), which makes the model pay less attention to repeated parts of the text and more to new information. They build a system that can do these updates quickly during the task, improving success rates on tests without much extra cost. Their results show aTTT helps keep the model's existing skills stable over long tasks rather than teaching it new abilities.
large language modelstest-time trainingpolicy driftself-trainingtoken reweightingn-gramsonline learningLoRAagent trajectoriesmodel adaptation
Authors
Yanbo Wang, Jinhua Hao, Yuze Shi, Kun Yuan, Ming Sun
Abstract
LLM agents often degrade over long episodes: as trajectories grow, they revisit explored states, repeat failed actions, and lose strategies that previously worked. Test-time training (TTT) offers a way to adapt model weights to the evolving task state, but existing LLM TTT methods largely adapt once to a fixed input. We study continuous TTT in multi-turn agent episodes, where each update changes the policy that generates later training text. This creates a self-training loop that helps when new trajectory information appears, but can amplify drift when the agent gets stuck and repeatedly trains on similar text. We find that update-text repetition distinguishes these regimes and introduce Agentic Test-Time Training (aTTT), a token-level reweighting method that downweights the loss on tokens appearing in repeated $n$-grams from prior updates while leaving novel tokens fully weighted. To run such updates inside live episodes, we build a concurrent serving system using vLLM's runtime LoRA API, limiting overhead to 1.9$\times$ the no-TTT cost. aTTT improves success by up to 5.0 points on ALFWorld and 4.9 points on SWE-bench Lite. The gains concentrate where models already have task competence but drift over long trajectories, suggesting that aTTT mainly preserves existing competence rather than teaching new abilities.