Agreement based measure captures ambiguity in rules and policies
A Contextual Measure of Ambiguity in Prescriptive Text
Computation and Language
Summary
Figuring out when rules or laws are unclear is tricky because the same words can be understood differently depending on the situation. The authors suggest measuring ambiguity by seeing how much different people disagree about how rules apply to various examples. They tested this idea using two different policy areas and found the measure matches real human disagreements and ignores unimportant text details. This method can also show how unclear a rule is linked to how much people support it.
What this means in practice
- •For legal compliance teams: Evaluate the clarity of contract clauses by measuring how much stakeholder disagreement arises across relevant scenarios.
- •For policy analysts: Identify ambiguous policies that may cause conflict by quantifying disagreement among different interpretations in context.
Authors
Sally Dong, Luke Thorburn, Andrew Konya, Michiel Bakker
Abstract
Ambiguity in prescriptive text such as laws, contracts, rules, policies, or recommendations can be both desirable (e.g., if it enables a law to adapt to changing context, such as new technologies) and undesirable (e.g., if it leads to arbitrary discretion or conflict over the law's interpretation). To help navigate these trade-offs, it would be valuable to have methods for quantifying ambiguity that are predictive of such outcomes, but existing measures capture how text is perceived in isolation rather than how it is interpreted in context. In this paper, we propose quantifying ambiguity as the degree of disagreement among a set of stakeholders ("raters") on how alternative prescriptive texts ("policies") apply in each of a set of relevant scenarios ("cases"). Using two policy domains, we present preliminary analysis suggesting that, even when using simulated raters, this approach (i) correlates with real human disagreement, and (ii) is robust to spurious text features. We then apply the measure to existing deliberation datasets to describe the association between policy ambiguity and support for those policies.