Atomic fact conditioned synthesis improves factual language model pretraining
AFCS: Atomic-Fact-Conditioned Synthesis for Factual Knowledge Pretraining
Computation and LanguageArtificial Intelligence
Summary
Language models need lots of factual knowledge, and one way to get that is by generating synthetic training data. Current methods rewrite passages or generate question-answer pairs but often miss exact facts like dates or item lists. The authors propose AFCS, a method that extracts precise atomic facts from source texts to guide data generation, making the synthetic data more factually accurate and detailed. When tested, this approach helped models perform better on factual question-answering tasks than previous methods.
What this means in practice
- •For nlp model developers: Improve factual accuracy of language model pretraining data by using atomic fact extraction to guide synthetic corpus generation.
- •For knowledge base maintainers: Augment knowledge bases with high-precision synthetic facts generated from structured document extracts for better coverage.
Authors
Yunhao Wang, Feng Zhang, Tinghao Yu, Xing Wu, Jiang Zhou
Abstract
Synthetic data is increasingly used in language-model pretraining. Existing knowledge-corpus synthesis pipelines, including WRAP and Nemotron-CC, usually take a source passage or document segment as input and generate rewritten or QA-style samples from that text. This can improve surface form and prompt diversity, but it does not explicitly require fine-grained source facts -- especially secondary entities, list/table entries, and precise temporal attributes -- to appear in the synthesized corpus. We propose AFCS (Atomic-Fact-Conditioned Synthesis), a pipeline that first extracts source-supported atomic facts and then uses each retained fact to drive generation. AFCS uses structure-aware extraction over the document tree, source-grounded verification, local context expansion, and multi-format generation. In continued-pretraining experiments with OLMo-3-7B and a 32B base on the same FineWiki source corpus, AFCS outperforms WRAP and Nemotron-CC on SimpleQA, SimpleQA Verified, PopQA, and HotpotQA. Granularity ablations show that fact-level synthesis improves over chunk-level, paragraph-level, and entity-level synthesis; and a 100B-token mid-training integration improves factual benchmarks and raises the 12-task general benchmark average by +1.09.