Data driven methods improve measuring quality of parliamentary talks
From DQI to C-DQI: Automatic Codebook Generation and Evaluation for Measuring Computational Discourse Quality Index
Computation and Language
Summary
Evaluating how well politicians discuss important issues in parliament is usually done by experts using fixed lists of rules, which can be slow and biased. The authors show that large AI language models can create and test new flexible lists of rules from actual parliamentary talks to judge quality. They find that these new data-driven rules agree better across different AI models and still match expert judgements. This could help analyze political discussions more easily and fairly on a big scale.
What this means in practice
- •For political data analysts: Automatically generate and assess coding schemes for evaluating political speeches and debates using AI models trained on real parliamentary data.
- •For social media monitoring teams: Adapt discourse quality measures generated by AI to analyze and track the quality of public political conversations online.
Tested on one dataset.
Authors
Esther Shizgal, Ron Keinan, Daniel Shtauber, Roee Tzvi Esquira, Ori Aronson, Matan Toledano, Or Rappel-Kroyzer, Liron Lavi, Renana Keydar, Ittai Bar-Siman-Tov, Reut Tsarfaty
Abstract
Effective democratic deliberation is vital for civic engagement and the health of legislative institutions. However, assessing parliamentary discourse quality is traditionally done using expert-scored, theory-based codebooks (CBs), which are rigid, labor-intensive, and prone to biases. While Large Language Models (LLMs) offer a scalable alternative for both schema induction and discourse annotation, ensuring schema quality and mitigating models' inherent bias remains a clear challenge. In this work, we propose a complete protocol for LLM-based codebook generation and evaluation. Starting from the widely accepted discourse quality Index (DQI), we compare top-down (theory-based), bottom-up (data-driven), and hybrid approaches for generating context-adapted schemas. To evaluate these schemas, we define an extrinsic proxy measure, validated by manual annotations, alongside an intrinsic inter-model agreement (IMA) metric, which leverages differing model biases to signal overall reliability. Our experiments on 100 Israeli parliament transcripts reveal that inductive, data-driven schemas achieve higher IMA than top-down methods while maintaining validity, demonstrating their potential for reliable, large-scale analysis of political discourse.