CORTEX: High-Quality Cross-Domain Organization of Web-Scale Corpora through Ontological Corpus Graph
2026-06-29 • Computation and Language
Computation and Language
AI summaryⓘ
The authors introduce Cortex, a new system that improves how large amounts of web text data are organized for training language models. Instead of just collecting flat lists of documents, Cortex arranges information in a structured way using something called an Ontological Corpus Graph, which links data by topics and quality. They also created CortexBench, a test to check how well different language models can search and reason with this organized data. Their experiments show that this approach helps improve data quality and usefulness for training. The authors plan to share their code, data, and benchmark publicly.
large language modelscorpus constructionOntological Corpus Graphontologyautomated evolutioncross-domain alignmentdata qualitybenchmarkLLM evaluation
Authors
Chengtao Gan, Xiaoke Guo, Yushan Zhu, Zhaoyan Gong, Zhiqiang Liu, Songze Li, Huajun Chen, Wen Zhang
Abstract
The continuous evolution of large language models drives escalating demands on data scale and quality, and as different training stages impose increasingly tailored data requirements, systematic organization of high-quality corpora becomes indispensable. Existing corpus construction pipelines confine the resulting corpora to flat, undifferentiated document collections, universally lacking systematic knowledge organization. We present Cortex, to our knowledge the first framework that elevates web-scale corpus construction from flat document filtering to structured knowledge organization through an Ontological Corpus Graph (OCG), a three-layer heterogeneous structure unifying a quality-refined content layer, a hierarchical lightweight ontology layer via LLM-driven automated evolution, and a cross-domain alignment layer enabling inter-domain association at arbitrary taxonomic resolution. Comprehensive experiments confirm the effectiveness of Cortex. In particular, we leverage the OCG to synthesize CortexBench, a cross-domain search-and-reasoning benchmark whose evaluation across eight frontier LLMs validates the effectiveness of quality refinement, domain organization, and cross-domain data synthesis. We will publicly release the complete codebase, a 24.14B-token refined corpus with its OCG, and CortexBench.