UnexpectedParrots improves behavioral study design before human testing
UnexpectedParrots: Iterating Behavioral Experiment Design with In-Silico Simulations
Human-Computer Interaction
Summary
Behavioral experiments often take a lot of time and money, and mistakes in design can only be spotted after collecting data from people. The authors created UnexpectedParrots, a tool that uses computer simulations representing human participants to find problems in experiments before real participants are involved. This tool looks at both answers and explanations from these simulated participants to give detailed feedback on study designs. Researchers who tested the tool found it saves time and helps think of new ideas, even if the simulations aren't perfect.
What this means in practice
- •For user experience researchers: Use simulated participants to identify and fix flaws in study designs before expensive human trials start.
- •For product development teams: Test early behavioral research ideas via simulations to save time and expand experimental approaches before real user testing.
Authors
Sanchaita Hazra, Bodhisattwa Prasad Majumder, Pao Siangliulue, Peter Clark, Amy X. Zhang, Joseph Chang
Abstract
Running behavioral experiments is expensive, slow, and prone to design flaws, often detectable after collecting data. We introduce UnexpectedParrots to iteratively refine behavioral study designs before a researcher recruits costly human participants. UnexpectedParrots relies on reasoning traces in addition to responses from LLM participants to uncover diagnostic critiques of structural errors and missed opportunities in existing study designs. When evaluated against a human-pilot or a no-simulation design-only baseline, UnexpectedParrots adds most value, producing unique, high-quality critiques. In a moderated user study with 12 proficient researchers, we find UnexpectedParrots useful, time-saving, and thought-provoking. Critiques provided by UnexpectedParrots often converge on issues researchers identify only during independent human pilots or additional formative interviews, even when the fidelity of the simulated populations remains questionable. Our users deem UnexpectedParrots useful not only as a tool to conduct early pilot studies and identify errors but also as an ideation tool to expand one's research horizon.