Metric rates individual skill across varied tasks without direct competition
All against the machine: the Solo score for rating skill in variable environments
Machine Learning
Summary
Many games and apps have people doing different tasks that vary a lot and have some chance involved, without any direct rivals. The authors created a way to score each person’s performance fairly, no matter the task’s difficulty or randomness. Their scoring method makes it easy to compare skill across many different situations. They tested it on hundreds of thousands of players in popular mobile games and showed it can reliably tell who is better or worse over time.
What this means in practice
- •For mobile game developers: Identify and track player skill levels reliably across diverse game challenges and progression stages.
- •For finance analytics teams: Assess individual trader performance fairly despite varying market conditions and random factors.
Authors
David Reguera, Xavier R. Hoffmann, Irene Pérez, Pol Colomer-de-Simón, Miquel Masoliver, Xavier Guardiola, Jan Wedekind, Marián Boguñá
Abstract
We propose a distribution-free metric to rate individual skill in ``player-versus-environment'' settings, where participants face heterogeneous tasks without direct opponents. Such settings are common in digital platforms, games, education, finance, and the benchmark evaluation of AI agents. They combine high randomness, tasks of widely varying difficulty, and unknown heterogeneity across individuals. Our metric maps each task outcome to a bounded performance score with zero population mean and variance bounded by $1/3$, regardless of the outcome distribution, so that scores are directly comparable across tasks. Aggregating these scores over tasks yields an interpretable skill score for each individual (the ``Solo'' rating), and a reshuffling null model tests whether that score exceeds what chance alone would produce. We validate the approach on the progression of $2\times10^5$ players in the mobile games Candy Crush Saga and Bubble Witch 3 Saga. The metric identifies high- and low-skill players with high statistical confidence, their classification persists over hundreds of subsequent levels, and a windowed version of the score tracks changes in performance along progression.