LongForge improves long-context data quality using reasoning graphs
LongForge: Measuring and Improving Long-Context Instruction Data via Evidence-Assertion Reasoning Graphs
Artificial Intelligence
Summary
Long context instruction data often lacks clear quality checks, making it hard to know which part of the information supports an answer and how many reasoning steps are involved. The authors created LongForge, a tool that analyzes this data by building graphs that link questions, answers, and evidence in documents. This analysis measures aspects like how broadly and deeply the reasoning covers the document and whether the answer is correct and complete. Then, LongForge uses this analysis to generate better training data, leading to improvements in tasks that require understanding long documents.
What this means in practice
- •For machine learning engineers: Improve training datasets for long-document understanding models by controlling reasoning depth and evidence coverage during data generation.
- •For natural language processing practitioners: Create more auditable and controllable instruction tuning data enabling better evaluation and debugging of complex question-answering systems.
Authors
Ziyang Chen, Junlong Jia, Xing Wu, Tinghao Yu, Songlin Hu
Abstract
Existing quality control for long-context instruction-tuning data is a one-shot judgment inside the generation pipeline that leaves only a scalar, revealing neither which document parts an answer uses nor how many reasoning steps it writes down; the data can thus be neither audited nor controlled. We formalize data quality as a property of the alignment structure among document, question, and answer, and build LongForge, which pairs an analyzer that measures long-context data with a synthesizer that generates it. The analyzer reverse-reconstructs each sample into an evidence-assertion reasoning graph and programmatically computes four dimensions on it: breadth, depth, correctness, and completeness. Every reading traces back to specific evidence spans and reasoning steps, making data auditable. Measuring seven mainstream datasets reveals two previously unquantified defects: breadth barely grows with document length, and depth is generally low, the two being nearly orthogonal. The synthesizer then embeds the analyzer into generation, gating on correctness and completeness and deferring breadth and depth to a training-time mixture over measured values, which makes the training distribution controllable. On two backbones, its two mixtures, LongForge-1 and LongForge-N, outperform the strongest baseline by 2.2-4.7 points on average over six long-context benchmarks, and in a controlled comparison that holds the gate and the candidate pool fixed, raising breadth and depth alone is worth a further 1.16 points, making the structural distribution of training data a lever on downstream quality rather than only a description of it.