Papers for

legal case analysts

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

Human AI working better without ground truth labels

Complementarity without ground truth?

Abstract: Human-AI complementarity captures the idea that a human assisted by an AI system can outperform both the human and the AI acting alone. In prediction tasks, this improvement is assessed against ground truth labels produced through a reference construction procedure and intended to represent the state of affairs to be predicted. Many human-AI interactions, including those where medical AI provides treatment recommendations, however, concern tasks other than predicting what obtains. Their relevant output concerns adequacy relative to a practical goal and depends on several evaluative dimensions. In this work, we examine the ground truth dependence of complementarity and extend it to human-AI interactions where ground truth is not the appropriate reference. We show that tasks beyond prediction can still admit a reference construction procedure that generates adequacy labels. We develop a route for extending complementarity to such settings. It replaces ground truth labels with context-dependent adequacy labels generated through a two-step scoring methodology inspired by multi-criteria decision analysis frameworks in healthcare and it makes the normative choices underlying reference construction explicit. Because the construction of adequacy labels embeds evaluative choices, we draw methodological inspiration from computational reliabilism to identify technical, scientific, and social factors relevant to supporting their construction. Our approach expands the empirical study of human-AI complementarity beyond prediction tasks, enabling its quantitative evaluation in decision-support settings where ground truth is not the appropriate evaluative reference.

Sat 7 NovArtificial IntelligenceHuman-Computer Interaction
The gist
Usually, when humans and AI work together, we measure how well they do by checking their answers against a known correct answer called ground truth. But sometimes, especially in fields like medicine, there isn’t a clear right answer to check. The authors explore how to measure human-AI teamwork in these situations by creating new ways to judge how good their decisions are based on the specific goals and context. They develop a method that scores decisions using multiple criteria, making the evaluation clearer and more suited to real-world uses where outcomes are complex.
Open → 2611.10893v1