Large language models tested for compliance with human rights standards
UDHRBench: Benchmarking Human Rights in Large Language Models
Computation and LanguageArtificial Intelligence
Summary
Language models like ChatGPT are used in many important areas, but how well they respect human rights is not well known. The authors created UDHRBench, a big test made of questions covering all parts of the Universal Declaration of Human Rights, to check how different models respond. Experts reviewed thousands of answers and found that models often break rights unintentionally by using logical but misaligned reasons like efficiency or risk. They also designed a way to have language models judge other models' answers automatically, with strong accuracy.
What this means in practice
- •For ai developers: Use UDHRBench to measure and improve their models’ adherence to human rights during development and deployment.
- •For ethics compliance teams: Automatically detect human rights misalignments in language model outputs using the LLM-as-a-Judge evaluation method.
- •For customer support platforms: Integrate automated human rights checks to ensure AI-generated responses align with ethical standards in user interactions.$Commercial implications: Enables compliance-focused AI tools for commercial customer service applications requiring human rights safeguards.
Authors
Alak Raad, Noreldin Morssy, Zeerak Suhail, Serin Atiani, Mustafa Jarrar
Abstract
Large language models are increasingly deployed in contexts requiring adherence to ethical and legal norms, yet their alignment with internationally recognized human rights standards remains underexplored. We introduce UDHRBench, a large-scale benchmark for operationalising the evaluation of large language model compliance with the Universal Declaration of Human Rights. The dataset comprises 1,124 human rights confronting questions covering all 30 articles and 4,496 responses from four large language models (ChatGPT, Grok, Gemini, and Fanar), manually annotated by experts for alignment, with high inter-annotator agreement ($κ= 0.801$). Our analysis reveals systematic and non-trivial misalignments, showing that violations typically arise through seemingly rational reasoning: appeals to efficiency, risk management, or scientific neutrality, rather than explicit rejection of rights. We further propose an LLM-as-a-Judge framework for automated evaluation using zero-shot and few-shot prompting, achieving 89.8\% weighted F1-score in detecting human-rights misalignment, alongside a novel Alignment Leniency Score that quantifies how judge models resolve ambiguous cases. The dataset, annotations, and code will be released as open-source resources (https://sina.birzeit.edu/UDHRBench).