Review prompts increase ratings but reduce score sensitivity in games
Ask More, Hear Less: How a Review Prompt Changes What Online Ratings Show
Human-Computer InteractionComputers and Society
Summary
Online platforms often ask customers to write reviews to get more ratings, but this can change what those ratings actually mean. The authors studied a video game store called Steam that started asking players for reviews after about three hours of play. They found that this prompt greatly increased the number of reviews and made the average rating more positive, but each individual review contributed less information to the overall score. This means the ratings still know about game quality but show it in a way that needs to be read differently than before.
What this means in practice
- •For platform product managers: Adjust rating labels on platforms that prompt users for reviews to accurately reflect changes in what scores represent.
- •For game developers: Interpret user ratings considering that review prompts increase positive reviews but decrease individual review impact on scores.
Tested on one dataset.
Authors
Zihan Chen, Lei Nico Zheng, Di Zhu
Abstract
Platforms ask customers for online reviews to reduce self-selection, but theory conflicts on whether review solicitation makes online ratings more or less informative. Selection theory implies that asking helps, voice theory that it dilutes the news, and research on rating inflation that compression destroys information. We reconcile these views with signal detection theory, which separates what a rating knows (its sensitivity) from what it shows (its criterion). We test the framework on Steam, a leading PC video game store, whose client began in October 2019 to prompt players for a review after about three hours of play. Using a difference-in-discontinuities design at that inferred threshold in 21.6 million Steam reviews of 315 games, we find that review counts just above the threshold become 4.25 times those just below, and the jump in the positive share grows by 6.6 percentage points. Agreement with critics changes little near launch and erodes later. The rating's movement changes far more: pass-through from reviews written in the first two hours falls from 0.73 to 0.18 just above the threshold, against 0.81 to 0.64 just below, and a fixed rating tier responds to 12% of downturns in those reviews above the threshold, down from 42%, against 41% and 47% below it. What the stream knows changes less. At equal review counts its sensitivity falls more above the threshold, because added reviews respond about half as much and scores near the ceiling are noisier per review, but at natural counts added volume offsets this loss, leaving a change of -0.03 relative to the stream below (95% CI -0.13 to 0.06). Failures that developers acknowledge, which run through complaints, still move the score after launch. Review prompts therefore mainly change how a rating must be read, and platforms that solicit reviews should recalibrate their rating labels.