Panel platform makes human tests for generative AI easier and private
PANEL: An Open-Source, Self-Hosted Web Platform for Human Evaluation of Generative Models
Human-Computer InteractionSound
Summary
Evaluating AI that creates text, images, or audio requires human judgement, but tools for collecting this feedback are limited or not private. The authors made PANEL, a free and self-hosted website where researchers can create and run evaluations using many types of media and questions. It includes features like skipping irrelevant questions, checking participants’ attention, and making results easy to analyze. PANEL respects privacy laws and lets participants withdraw easily. It’s available online for anyone to use and improve.
What this means in practice
- •For ai development teams: Set up private studies comparing your own generative models with human raters without relying on commercial survey platforms.
- •For user experience researchers: Run interactive, multimedia human tests with fine control over question flow and data analysis in any research setting.
Authors
Matteo Spanio, Andrea Poltronieri, Martín Rocamora
Abstract
Human judgement is the reference measure for evaluating generative models, yet the software used to collect it lags behing the methodology. Researchers adapt listening-test frameworks designed for perceptual protocols such as MUSHRA, rely on closed commercial survey platforms, or implement single-use web applications. Live arenas such as Chatbot Arena and Music Arena rank publicly deployed systems at scale, but do not support controlled comparisons of a laboratory's own models with its own participants. We present PANEL, an open-source, self-hosted platform for such studies. A study is authored in the browser and distributed as a single link, with audio, video, image, and text stimuli, seven question types, and screening and skip logic. The platform reports per-question summaries, across-condition significance tests, pairwise win rates and Bradley--Terry scores, and supports power analysis from pilot data. Consent versioning, self-service withdrawal, retention enforcement, and audit logging support GDPR-compliant operation. Each study exports as a machine-readable specification. PANEL is available at https://github.com/matteospanio/panel.