Papers for

social science analysts

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

Human and AI work together to improve qualitative coding accuracy

Scaling Qualitative Analysis with Large Language Models: Development and Validation of the Smart Coder Framework for Human-LLM Collaborative Coding

Abstract: Current uses of large language models (LLMs) for qualitative coding often rely on ad hoc prompting with limited guidance for reliability and human oversight. To address these limitations, we propose the Smart Coder Framework, a three-phase approach structuring human-LLM collaborative coding through Instruction, Prompt Strategy, and Reliability Check. We evaluated the framework using 72 middle-school mathematics teacher interview transcripts and a human-consensus benchmark. A factorial design compared instructed versus uninstructed prompts, few-shot versus few-shot plus chain-of-thought prompting, and single-pass versus self-consistency coding. Twelve transcripts were used for prompt development and calibration, and 60 for the formal evaluation. In the results, instruction consistently improved agreement. Chain-of-thought further improved agreement under adequate instruction, and self-consistency produced smaller, theme-dependent gains but helped identify unstable decisions for human review. Performance also varied by transcript characteristics. Overall, the framework systematically structures and monitors LLM-assisted qualitative coding while preserving human authority over coding decisions.

Fri 23 OctArtificial Intelligence
The gist
Qualitative coding is a way to organize and interpret complex text data, but doing it with AI has often been unreliable and inconsistent. The authors developed the Smart Coder Framework, which guides humans and AI to work together in three clear steps to make coding more reliable. They tested their approach on real teacher interviews and found that giving clear instructions and thoughtful AI prompting improved agreement with human consensus. This framework helps keep humans in control while using AI to speed up analysis.
Open → 2610.33205v1