The Announcement Carries the Cue: Markup, Boundaries, and the Notation of Pre-Training Corpora
2026-08-10 • Computation and Language
Computation and LanguageArtificial Intelligence
AI summaryⓘ
The authors study how the way documents are formatted and annotated affects language models, a factor usually not recorded in datasets. They introduce 'clean-window survival' to measure how much text still requires the model to guess boundaries after text extraction. Their experiments show that removing structure cues makes text harder to predict, but changing the exact notation doesn’t affect models much. They suggest focusing on the presence of structural announcements rather than their form, and recommend documenting extraction methods on data cards for better training and evaluation.
notationcorpuslanguage modelstext extractionboundary inferenceclean-window survivalstructural annotationdataset cardtext formattingmodel behavior
Authors
E. M. Freeburg
Abstract
How a document's arrangement is written down, its notation, is a training variable that no dataset card records. The field has established that text-extraction choices change model behaviour, and has never once measured the notation of what those choices put into the corpus. We define clean-window survival, a deterministic count of how much of a stream still demands the boundary inference, and measure notation on three fronts. What corpora carry: a census of thirteen public corpora, where survival falls to 0.153 in a vision-converted PDF slice against 0.889 in C4; the scarce resource is not unmarked text but long unmarked text; a pre-registered supply test finds what remains institutional, not consumer. Our own pre-registered prediction failed: converters do not fabricate structure on prose, and that null forced the reliability mechanism that survives it. What readers use: across five base models spanning 0.6B to 8.2B and two pipelines, deleting a structural announcement makes the following prose measurably harder to predict, while swapping its notation moves nothing. That zero does not make notation unimportant; it relocates the variable: the operative cue is the announcement, not the sigil. What writers impose: a bounded null. Base models do not impose the marked register above the authored baseline, and handed prose with every announcement deleted they do not put one back, at a rate indistinguishable from zero against an authored reference of zero. We ship the format those measurements imply: the pure frame, paragraphs in authored order, every announcement deleted into a reversible sidecar, mixed against the marked copy over announcement presence rather than notation. Choose format operators by the capability they train, not by the fidelity they preserve, and record extractor identity and survival on data cards.