Listwise interfaces improve human feedback in optimization tasks
Rethinking Opinion Measurement Interfaces for Human-in-the-Loop Optimization
Human-Computer InteractionMachine Learning
Summary
Human-in-the-loop optimization means a system tries to learn what people prefer by asking them to make choices or give ratings over time. However, usual ways of collecting these opinions, like rating scales or picking preferred options, can lead to inconsistent or incomplete feedback. The authors studied six different ways to collect opinions on text and image tasks from 360 people. They found that showing a list of options and asking for feedback on all at once works better than single ratings or choices. This approach helps the system learn more and makes the feedback more reliable.
What this means in practice
- •For product designers: Design customer feedback tools that collect ranked lists to improve iterative product personalization.
- •For ux researchers: Create study interfaces that use listwise preference measurement to gain richer, more consistent user feedback.
Authors
Changkun Ou, Sven Mayer, Daniel Buschek, Andreas Butz
Abstract
Human-in-the-loop optimization refers to systems that present choices to a human decision-maker and query their judgment to optimize system parameters iteratively. The user interface typically provides either an absolute rating scale or asks for a preferred option. However, these approaches can potentially suffer from calibration problems, inconsistencies, and completeness violations of user preferences in such a sequential evaluation. This paper revisits the design space of opinion measurement interfaces for human-in-the-loop optimization. We conducted a study (N=360) using text summarization and image color enhancement tasks to evaluate human sequential feedback in six representative opinion measurement interfaces. Based on our analysis, we recommend using a listwise approach to support sequential opinion measurement, which counterweighs individual rating and preferential choice limitations and conveys more information for subsequent use. We further discuss the trade-offs in different user interface designs and provide guidelines to inform future research.