Open agents make spotting fake AI survey answers harder
Cheap, open agents make LLM pollution harder to mitigate
Artificial Intelligence
Summary
People use surveys to learn what humans think or do, but sometimes AI systems answer these surveys instead. The authors found that freely available AI agents running on your own computer can answer surveys just as well as expensive commercial ones without paying fees. These open agents are harder to detect because different detection tests catch different kinds of AI answers, and no single test works for all. Looking closely at free-text answers is the best way to tell AI from humans. This means more careful, layered checks are needed to keep surveys trustworthy.
What this means in practice
- •For data integrity teams: Use multilayered checks focusing on open-text analysis to better detect AI-generated survey responses and maintain dataset quality.
- •For ai tool developers: Benchmark open and commercial AI agents’ performance and detectability to design more reliable autonomous agents for real-world tasks.
Authors
Raluca Rilla, Anne-Marie Nussberger, Rui Mata, Dirk U. Wulff
Abstract
Large Language Model (LLM) pollution occurs when synthetic responses contaminate data intended to capture human behavior. High deployment costs have so far limited the risk posed by autonomous survey agents. However, open-weight models paired with open-source agentic frameworks may have removed this barrier. We compared the performance and detectability of nine agent configurations, ranging from fully open variants to closed commercial ones. Each agent autonomously completed a survey containing multiple response types yielding various detection checks. Fully open agents ran locally without usage fees and performed competitively with commercial alternatives. Open and commercial agents failed different sets of checks, and no single check reliably detected all agents, but open-text responses discriminated best between agents and humans. These findings identify fully open agents as a distinct risk for LLM pollution and support multilayered detection strategies emphasizing open-text analysis.