OctoLong: Mid-Training On Cross-Repository Code Contexts Enhances Long-Context Modeling
2026-08-05 • Artificial Intelligence
Artificial IntelligenceMachine LearningSoftware Engineering
AI summaryⓘ
The authors created OctoLong, a new way to gather very long and detailed code examples by connecting different parts of software using tools like parsers and package managers. They then used this data to train OctoLong-Instruct, a set of language models that can understand and work with much longer pieces of code than usual. Their experiments show that adding just a small amount of this specialized data helps the models perform better on tasks requiring understanding of long code dependencies and improves coding abilities even with shorter contexts. This work focuses on improving how language models handle long and complex code compared to prior datasets dominated by books and articles.
language modelscontext lengthAST parserpackage managercode dependenciesinstruction tuninglong-context learningself-improvementagentic workflowsrepository-level code understanding
Authors
Indraneil Paul, Falko Helm, Goran Glavaš, Iryna Gurevych
Abstract
Context lengths of language models (LMs) have dramatically increased, driven by the demands for in-context learning, self-improvement, and long-horizon agentic workflows. Existing long-context corpora, however, are dominated by books, academic articles, and code repositories, which are finite resources and often scarce in long-distance dependencies. In this work, we introduce OctoLong, a context engineering pipeline that instruments an AST parser, a language server backend, and a package manager to facilitate the recursive retrieval of code references, enabling the curation of dependency-rich code contexts of millions of tokens in length. We then train OctoLong-Instruct, a suite of capable long-context open LMs, derived from base models ranging in size from 600M to 14B parameters, via context-extension mid-training on a ~50B-token mixture containing ~6.2B tokens of OctoLong code contexts, followed by ~10B tokens of instruction tuning. Our training ablations and experimental evaluations against 18 state-of-the-art open-weight long-context LMs show that supplanting just 12% of traditional context-extension corpora with OctoLong data yields substantial gains in long-range retrieval, long-term state tracking, repository-level code understanding, and downstream agentic tasks, while also enhancing API usage in short-context coding scenarios.