Self-Guided Test-Time Training for Long-Context LLMs
2026-07-10 • Computation and Language
Computation and LanguageArtificial Intelligence
AI summaryⓘ
The authors studied how large language models (LLMs) handle really long texts but found that simply making the input longer doesn't always help and can even lower accuracy. They used a method called test-time training (TTT) that adjusts the model based on the test input, but adapting to the whole long text is too expensive and randomly picking parts to learn from can add noise. To fix this, they created Self-Guided TTT (S-TTT), where the model first picks the important parts of the text before training itself on just those parts. This approach improved performance on tough long-text reasoning tests by up to 15%.
large language modelslong-context processingtest-time trainingparameter adaptationevidence span selectionlanguage modelingLongBench benchmarksmodel accuracyinstance-specific trainingself-guided learning
Authors
Xinyu Zhu, Zhe Xu, Xiaohan Wei, Yunchen Pu, Fei Tian, Chonglin Sun, Kaushik Rangadurai, Hua Zhi, Frank Shyu, Sandeep Pandey, Luke Simon, Yu Meng, Xi Liu
Abstract
Long-context processing has become increasingly important for large language models (LLMs), but simply extending the context window does not guarantee effective utilization of long inputs. As input length grows, accuracy often degrades, indicating that models still struggle to identify and use the evidence most relevant to a question. A promising way to improve long-context utilization is test-time training (TTT), which treats the test context as a training example for instance-specific parameter adaptation. However, applying TTT to the entire long context is prohibitively expensive, while adapting on randomly sampled spans introduces severe noise. Because most spans in a long context are irrelevant to the specific question, training on them may even degrade the base model's performance. Our preliminary study shows that TTT is highly sensitive to training-span quality: on LongBench-v2, TTT on randomly sampled spans hurts performance, whereas TTT on oracle spans substantially improves it. Motivated by this, we propose a simple method, Self-Guided TTT (S-TTT): before adaptation, the model identifies the evidence spans it should learn from, and the standard language-modeling training objective is applied only to those selected spans. On two challenging long-context reasoning benchmarks, LongBench-v2 and LongBench-Pro, S-TTT improves accuracy for both Qwen3-4B-Thinking-2507 and Llama-3.1-8B-Instruct, achieving up to a 15% relative improvement.