Ai collaboration changes how science ideas are judged not quality
Beyond Better Ideas: How Human-AI Collaboration Shapes Scientific Ideas and Human Perceptions
Human-Computer Interaction
Summary
Scientists worked with AI tools to come up with new scientific ideas and studied how people judge these ideas. They found that working with AI doesn’t always lead to better ideas as rated by humans, but revising ideas repeatedly helps improve them. Interestingly, AI judges tend to prefer ideas created with AI assistance even when humans don’t see a quality boost. People who collaborate with AI find it useful but don’t always get faster results as they expect. The study shows AI affects not just the ideas but also how scientists think about and judge those ideas.
What this means in practice
- •For science research teams: Design workflows that focus on iterative idea improvement alongside AI collaboration to enhance scientific creativity without assuming AI always improves idea quality.
- •For ai system evaluators: Develop evaluation protocols that account for differences between human and AI judgments when ranking AI-assisted scientific ideas.
Authors
Qian Chen, Saatvik Pradhan, Janice Ahn, Berk Atil, Angkai Li, Shijie Lu, Nobin Sarwar, Michael Ulis, Ruyuan Wan, Wenpeng Yin
Abstract
As AI increasingly becomes a collaborator in scientific research, its impact may extend beyond improving ideas to reshaping how they are evaluated, attributed, and perceived by humans. Yet existing research has largely examined these dimensions in isolation. We present a unified empirical study of Human-AI scientific ideation, investigating two complementary perspectives: i) how AI collaboration shapes scientific ideas and their evaluation, and ii) how firsthand collaboration reshapes human perceptions of AI. Through a shared experimental setting involving Human-AI and Human-Human research teams, human and LLM evaluators, collaboration-provenance analysis, and participant surveys, we uncover several counterintuitive findings. First, AI collaboration shows no significant advantage in human-evaluated idea quality, while iterative revision consistently improves quality. Second, AI judges can approximate human scores yet disagree on idea rankings; notably, an AI judge favors AI-assisted proposals without a corresponding advantage in human evaluations. Third, collaboration provenance remains only partially detectable, revealing substantial overlap between Human-AI and Human-Human scientific outputs. Finally, firsthand AI collaboration is associated with greater perceived usefulness but smaller efficiency gains than expected, suggesting a nuanced recalibration of human expectations. Overall, our findings challenge the notion that AI's value in scientific ideation lies primarily in producing better ideas, revealing instead its broader influence on scientific creation, evaluation, attribution, and human perceptions.