New dataset enables identifying institutions in international law texts
IntLawNER: A Named Entity Recognition Dataset and Benchmark in International Law
Artificial IntelligenceInformation Retrieval
Summary
International law texts are important but lack detailed labeling of named entities like organizations and courts. The authors created IntLawNER, a dataset with nearly 3,000 sentences from international legal documents annotated with seven types of institution-related entities. They used a mix of automated and human review to build the dataset and found some challenges in how well machines and people agree on annotations. Testing shows that large language models improve when given a few examples, but still struggle with rare labels or types based on function rather than name. This resource helps future tools extract structured information from international law.
What this means in practice
- •For legal technology developers: Extract precise references to institutions and entities from international law documents using the IntLawNER dataset to train or test entity recognition tools.
- •For government policy analysts: Automatically identify and track mentions of international legal bodies and entities in large volumes of legal text to support policy monitoring.
Authors
Genis Skura, Roland Bouffanais, Didier Wernli
Abstract
International law provides the normative framework through which states coordinate action, regulate armed conflict, and protect human rights, yet its texts remain without token-level named entity recognition (NER) resources. We introduce IntLawNER, a NER dataset and benchmark for codified sources of international law, covering 2,987 gold-annotated sentences and 8,094 entity spans from International Court of Justice (ICJ) decisions, UN Security Council resolutions, and European Court of Human Rights (ECtHR) judgments, annotated with seven institution-specific entity types. We construct IntLawNER with a cost-effective hybrid algorithmic-agentic pipeline that reduces 468k source sentences to a compact annotation set through candidate retrieval, LLM-based vetting, and human review, with 89.6% of gold spans accepted unchanged from the silver layer. However, the silver-to-gold analysis reveals that human-machine aggregate agreement metrics can be misleading in domain-specific NER: Cohen's kappa=0.964 on boundary-matched spans masks a macro-F1 of 0.753 when missing entities, boundary errors, and label corrections are included. The benchmark shows that zero-shot span-based GLiNER collapses on entity types dependent on institutional function rather than surface form (0.243 micro-F1), while fine-tuned transformers struggle on rare labels. Carefully selected few-shot examples that demonstrate label contrasts improve every LLM over zero-shot prompting, with Claude Opus 4.6 reaching the best score of 0.873 micro-F1. We release IntLawNER as a benchmark and reusable resource for extracting references in international legal texts.