CONSISTRE: A Unified Consistency-Aware Framework for Document-Level Relation Extraction with Large Language Models
2026-07-27 • Computation and Language
Computation and Language
AI summaryⓘ
The authors address a problem in document-level relation extraction, where large language models often make inconsistent predictions about relationships between entities. They propose CONSISTRE, a method that improves consistency in two ways: one works during model use by refining predictions without retraining, and the other trains smaller models to learn consistent reasoning from bigger models. Their experiments show both methods reduce contradictions and improve reliability, with the training approach making smaller models nearly as good as powerful proprietary ones but cheaper to run. This helps ensure relationships identified across documents are more logically sound.
Document-level relation extractionLarge language modelsRelational constraintsTransitivitySymmetryKnowledge distillationReinforcement learningSelf-reflectionPromptingF1 score
Authors
Mingxuan Sun
Abstract
Document-level relation extraction (DocRE) aims to extract relations among multiple entities across extended contexts while maintaining consistency across predicted triples. Although large language models (LLMs) show remarkable reasoning capabilities in information extraction, their predictions are typically generated independently for each candidate triple and may violate fundamental relational constraints such as transitivity, symmetry, and functional uniqueness, leading to contradictory and unreliable outputs. We propose CONSISTRE, a unified consistency-aware framework for DocRE that addresses this limitation through two complementary tracks. The first operates at inference time for black-box LLMs, combining constraint-aware prompting, constraint-based verification, and iterative self-reflection to refine predictions without task-specific fine-tuning. The second injects consistency knowledge into smaller open-source models via a knowledge distillation and reinforcement learning pipeline: reasoning traces from a powerful teacher are distilled into a student via supervised fine-tuning, followed by GRPO alignment using a composite reward that jointly optimizes extraction performance and relational consistency. Together, the two tracks cover both API-accessible and locally deployable scenarios under a unified consistency formulation. Experiments on DocRED show that both tracks outperform their baselines, with the inference-time track achieving competitive F1 using off-the-shelf black-box LLMs and the training-time track substantially narrowing the gap between 7--8B open-source models and state-of-the-art proprietary LLMs at a fraction of their inference cost. Ablation studies confirm that explicit consistency modeling mitigates relational contradictions and enhances the reliability of LLM-based DocRE across both deployment paradigms.