Papers for

social science research teams

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

Human study programs run using configurable ai participants

Executable Human Study Programs with Silicon Participants

Abstract: We introduce HumanStudyProgram, a framework that represents a documented human study as an executable program with configurable LLM-based silicon participants. Its computational abstraction separates study logic from participant implementation through standardized APIs for presenting information and collecting responses. We implement a researcher-facing authoring tool that reconstructs source-linked study descriptions from papers and supplementary materials, supports researcher review, and generates executable HumanStudyProgram artifacts. Researchers can then configure silicon participants, run the programs, and analyze results from aggregate findings to individual responses. We demonstrate the approach through study cases and evaluate the tool through a hands-on expert review with 12 social science researchers. All completed the workflow and identified potential uses in understanding published protocols, prototyping studies, and examining AI behavior. We aim to improve the reproducibility of human studies by reconstructing their documented procedures as programs that others can reuse and build upon.

Mon 9 NovHuman-Computer Interaction
The gist
It can be hard to repeat or build on human studies because their procedures are written as text instead of code. The authors created a framework that turns a study’s description into a computer program that can run with AI-based participants simulating humans. This helps researchers check study methods, test ideas before real experiments, and analyze behaviors of AI in simulated studies. Social scientists tested the tool and found it useful for understanding and prototyping studies.
Open → 2611.11969v1

Multimodal models show varied accuracy on video analysis tasks

What Does the Model See? Representation Validity in Multimodal LLM-Based Video Analysis

Abstract: Multimodal large language models (MLLMs) allow social scientists to analyze video at scale, but researchers must decide which parts of a video to provide as input. We use the term representation validity to examine how text, audio, video, and combined inputs affect measurement and subsequent analysis. We compare input representations across six open-weight MLLMs and three human-annotated video datasets. No representation consistently performs best. On seven news-related constructs, native-video performance varies little across models (macro-F1 = .70-.77), while extracted-text performance ranges from .52 to .80. Three models perform better with text and three with video. Combining inputs rarely improves performance. Higher classification accuracy also does not reliably produce estimates closer to those based on human codes. The highest-F1 representation best reproduces human-coded regression coefficients in 17 of 42 comparisons. Even when each model uses its highest-F1 deployable representation, 45% of sample-prevalence estimates differ from the human-coded benchmark by more than ten percentage points. Input representation therefore requires validation for both the construct being measured and the quantity estimated from the resulting labels.

Tue 3 NovComputer Vision and Pattern RecognitionComputation and Language
The gist
When computers analyze videos to understand their content, they can look at the pictures, hear the sounds, or read text extracted from the video. This paper finds that no single input type—video, audio, or text—always works best for understanding videos. Combining these inputs does not consistently improve accuracy either. Even the best computer judgments can still differ quite a bit from human annotations, meaning it's important to carefully check how video understanding models are used depending on the task.
Open → 2611.05181v1

Systems capture web browsing with privacy safeguards for research

GESIS Surf: An Open-Source Infrastructure for Privacy-Preserving Longitudinal Web Browsing Data Collection

Abstract: Understanding how people navigate digital media environments requires behavioral data beyond survey self-reports. However, existing infrastructures for collecting such data reproducibly, sustainably, and with strong privacy guarantees remain limited: commercial solutions lack transparency and data governance, while academic tools are typically designed for narrow, short-term use cases and do not support general-purpose academic panel data collections. We address this gap with GESIS Surf, a research infrastructure for large-scale longitudinal collection of web behavioral data in naturalistic environments. The system consists of a browser extension (Chrome and Firefox), a backend with a REST application programming interface (API), and a relational database. It passively captures user interactions, page metadata, and full HTML snapshots of each rendered page, embedding each observation within a hierarchical session model that preserves the structure of browsing behavior. GESIS Surf implements privacy-by-design through strict opt-in participation, per-domain collection rules, and client-side data minimization, ensuring that sensitive data are controlled at the point of collection. The system has been deployed with more than 1,000 participants in a large-scale academic panel infrastructure that includes surveys of participants, and supports reproducible, content-level, and structurally rich analysis of digital behavior. By combining hierarchical session modeling, content-level capture, and integrated privacy enforcement within a unified system, GESIS Surf provides a general-purpose infrastructure for studying digital behavior in a reproducible manner over time.

Fri 30 OctCryptography and SecurityComputers and Society
The gist
Collecting accurate data about how people browse the web is important for understanding their behavior but is usually difficult due to privacy concerns and limited tools. The authors created GESIS Surf, a system that records web browsing details in a way that respects user privacy by letting people opt in and minimizing personal data collected. It works through a browser extension and stores structured browsing sessions, allowing detailed studies over long periods. This setup has been used with over 1,000 participants and supports research that can be repeated and verified.
Open → 2610.41441v1

PaperDoctor improves scientific paper feedback with evidence and action

PaperDoctor: Evidence-Grounded and Actionable Feedback for Scientific Papers in Progress

Abstract: Autoresearch agents are reshaping the research ecosystem, but they can also let flawed claims enter the literature at scale. Human advisors catch such issues in drafts through careful, traceable feedback, yet advisor-style assessment requires extensive manual effort and does not scale. To shift automated paper assessment from a judge to a diagnostician, we introduce PaperDoctor, an agent framework for pre-submission feedback with three key innovations. First, a holistic hierarchical framework evaluates writing, layout, references, code, theory, prior work, and experiments through three layers: L1 surface screening, L2 typed verifiers that route each claim to the appropriate evidence, and L3 reproducers that rerun experiments by priority. Second, each finding contains an observation, a pointer to specific evidence such as a sentence, equation, or code line, and a revision suggestion, making critiques auditable and actionable. Third, PaperDoctor selectively rebuilds and reruns experiments based on claim importance and compute budget, surfacing reproducibility gaps and quantitative limitations that are invisible from the manuscript alone. We evaluate PaperDoctor on 30 in-progress papers, yielding 70.6% agreement and all positive holistic scores, and on 40 manuscripts across machine learning, natural science, and social science, covering human- and AI-authored papers with code. Overall, PaperDoctor produces more auditable feedback than human and other agentic reviewers, pairs critiques with concrete suggestions by design, and complements dimensions often overlooked by human reviewers. We also develop an interactive interface that lets authors browse findings grounded in their paper. PaperDoctor reframes automated paper assessment as diagnosis rather than verdict, taking a concrete step toward AI advisors for more rigorous AI-assisted scientific discovery.

Tue 15 SeptComputation and LanguageMultiagent Systems
The gist
Checking scientific papers before they are published can be slow and hard to do well with only humans. The PaperDoctor system helps by automatically finding problems in a paper’s writing, experiments, and references. It points to exact parts of the paper or code where issues appear and suggests how to fix them. This helps scientists improve their papers better and faster, making research clearer and more reliable.
Open → 2609.16995v1

AI coders debate and agree to improve qualitative coding accuracy

How AI Coders Discuss, Disagree, and Reach Consensus: Challenges and Opportunities for LLM-Based Qualitative Coding

Abstract: The utility of AI in multi-coder qualitative coding has been widely discussed, yet little empirical evidence exists to delineate the contexts in which it performs reliably. We address this gap by quantifying the effectiveness of multi-agent LLM coding across varied qualitative datasets, revealing key contextual and structural factors that mediate coding outcomes. We developed a literature-informed baseline pipeline that enables AI agents to independently code, debate, and reconcile disagreements. Results revealed that coding accuracy depends on factors such as codebook length, qualitative data similarity, and agent disagreement. Notably, intense and unresolved debates between agents led to higher accuracy. Our analysis showed that while LLMs emulate many human discussion behaviors, they lack adaptive responsiveness to context. From these findings, we offer design recommendations for building automated coding systems. Our open-source AI discussion dataset and methodological framework lay the groundwork for advancing the design of AI-mediated automated thematic analysis.

Thu 10 SeptHuman-Computer InteractionArtificial Intelligence
The gist
Figuring out how AI can help people label text data is tricky because it depends on many things. The authors studied how multiple AI agents code, argue, and come to agreements when labeling text from different sources. They found that longer instructions, how similar the data is, and how much the AI agents disagree affect how well the AI does. Surprisingly, harder debates between AI agents sometimes meant the final answers were more accurate. The authors also noticed AI behaves a bit like humans in discussions but doesn't adapt well to changing contexts.
Open → 2609.11109v1