Model learns to handle long texts with small context windows

Trained Agentic Context Management

Machine LearningComputation and Language

Summary

Reading very long texts is hard for language models because they need to remember lots of information at once. The authors found a way to train a model that can manage long documents by teaching it to call itself with smaller pieces of the text and read parts as needed. They trained a model this way on fake but varied text, and it became as good as a much bigger model that uses huge amounts of text at once. This means smaller models can work with very long texts effectively.

What this means in practice

  • For ai developers: Create language models that efficiently process very long documents using limited context tokens by training models with self-invoking and selective reading tools.
  • For content management teams: Handle and analyze large bodies of text in automatic tools without needing extremely large context windows in the underlying AI models.

Authors

Bryce Sandlund

Abstract

We study long context language models. Instead of training long context natively, or designing a long context harness, we train a model over the simplest possible harness: a tool to call itself with any specified prompt and a tool to read tokens in a range from the input context. We finetune Qwen3.6-35B-A3B on a diverse synthetic dataset using this harness. With only 8,000 tokens of context, our small model is as strong as GPT-5.4 with 1M tokens of context on the OOLONG-synth benchmark when document length exceeds 40K tokens.