Hebrew relation extraction benchmark created with multi language AI denoising
HeRE: A Novel Benchmark for Hebrew Relation Extraction via LLM-Guided Denoising of Knowledge Graph Alignments
Computation and LanguageMachine Learning
Summary
Extracting information about relationships in text is harder for languages with fewer resources, like Hebrew. The authors developed a smart way to clean up noisy data by checking if sentences truly support specific facts using language models. They tested different methods and found models that work well with Hebrew and multilingual data. Using this approach, they built HeRE, a huge new dataset to help computers better understand relationships in Hebrew texts, opening doors for further research and applications.
What this means in practice
- •For natural language processing engineers: Use the HeRE dataset and pipeline to train and evaluate models extracting relationships from Hebrew text with more accuracy.
- •For multilingual ai developers: Adopt the authors’ language-agnostic denoising method to build relation extraction datasets for other low-resource languages efficiently.
Authors
Ron Keinan, Amir David Nissan Cohen, Reut Tsarfaty
Abstract
While LLMs demonstrate strong capabilities in English Relation Extraction (RE) tasks, a significant gap remains in assessing their RE performance in low-resource languages, due to a lack of standardized benchmarks. Creating native, language-specific annotated RE benchmarks is expensive and requires human expertise, while relying on translation yields suboptimal, inaccurate datasets that introduce English-centric biases. To tackle this, we propose a scalable, language-agnostic methodology to create RE data by denoising naturally occurring yet noisy alignments of knowledge bases and corresponding texts. We define a proper alignment via a Natural Language Inference (NLI) task, where a relation triplet is entailed by an input text. We empirically assess five denoising strategies, and demonstrate that multilingual models outperform language-specific models in the few-shot LLM setting, while Hebrew-specific encoders remain competitive under NLI fine-tuning and are strongest in the label-free cross-training setting. We further find that prompting in the native language achieves performance on par with or slightly exceeding English prompts, confirming that the pipeline operates naturally in the target language. The best open-weights model, Gemma-4-31B-it (F1\,=\,0.915), matches the best API model, GPT-5.4 (0.921), and reaches F1\,=\,0.844 on a fully held-out, dual-annotated test set. Applying the optimized pipeline, we release HeRE: a novel, massive, 1.5-million-instance silver dataset for Hebrew RE, designed to support data-intensive experimentation and model fine-tuning. HeRE bridges a critical gap in standard Hebrew RE evaluation, while providing the NLP community with a reproducible, language-agnostic pipeline for RE dataset construction in other low-resource settings.